diff --git a/.env.example b/.env.example index 86d35a7b..3cb58e45 100644 --- a/.env.example +++ b/.env.example @@ -109,6 +109,10 @@ KERNEL_ENV=base # it from the machine: its cores and memory divided by a new submission's # default cores and memory. The panel's Resources tab overrides this. # COMPETITION_SLOTS=auto +# Entries of KERNEL_GPUS given to competition submissions instead of rooms: +# COMPETITION_GPUS= +# Submissions sharing one such card (1-8), each held to its share of memory: +# COMPETITION_GPU_PER_DEVICE=1 # Local Docker room network only; broker jobs use Kubernetes NetworkPolicy. # KERNEL_NETWORK=colloq # Every room container runs hardened: uid 1000, no capabilities, diff --git a/Makefile b/Makefile index 381ca6a7..ba23eed5 100644 --- a/Makefile +++ b/Makefile @@ -663,11 +663,14 @@ TAG = colloq-vast:dev .PHONY: vast-vm vast-image vast-image-run vast-image-stop -vast-vm: ## Rent a Vast VM that boots the colloq-vast image. HOST=name GPU="RTX 3070" CPU=48 RAM=96 OFFER=number INSTANCE=id +vast-vm: ## Rent a Vast VM that boots the colloq-vast image. HOST=name GPU="RTX 3070" GPUS=2 GPU_RAM=24 CPU=48 RAM=96 DISK=100 MAX_PRICE=1.5 OFFER=number INSTANCE=id @# Over the API, because only a rent request with "vm": true makes a VM; @# the console's template starts even vastai/kvm as a Docker instance. The - @# on-start is backups/vast-onstart.sh (ONSTART=… for another one). - @NAME="$(NAME)" HOST="$(HOST)" GPU="$(GPU)" CPU="$(CPU)" RAM="$(RAM)" OFFER="$(OFFER)" INSTANCE="$(INSTANCE)" ./scripts/vast-vm.sh + @# on-start is backups/vast-onstart.sh (ONSTART=… for another one). Empty + @# variables mean "not asked for": DISK and MAX_PRICE then come from .env, + @# as before, and GPUS and GPU_RAM filter nothing. + @NAME="$(NAME)" HOST="$(HOST)" GPU="$(GPU)" GPUS="$(GPUS)" GPU_RAM="$(GPU_RAM)" CPU="$(CPU)" RAM="$(RAM)" \ + DISK="$(DISK)" MAX_PRICE="$(MAX_PRICE)" OFFER="$(OFFER)" INSTANCE="$(INSTANCE)" ./scripts/vast-vm.sh vast-image: ## Build the colloq-vast image. TAG=name:tag PLATFORM=linux/amd64 @# The version comes from the root package.json (its only source, see diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 5da6a42c..950b998f 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -108,9 +108,10 @@ Room kernel images install Python packages from PyPI at build time: - `kernel/requirements.txt`: Jupyter Server, ipykernel, NumPy, pandas, Matplotlib, Plotly, scikit-learn, requests and Black; -- `kernel/environments/*.txt`, for example seaborn, PyTorch (the CPU build in - `cv`, the CUDA build in `base-gpu`), torchvision, timm, OpenCV and - Transformers. +- `kernel/environments/*.txt`, for example seaborn, CatBoost, LightGBM and + XGBoost (`kaggle-base`), PyTorch and torchvision (the CUDA build in + `base-gpu`), and OpenCV and timm (`kaggle-gpu`). Each package keeps its own license inside the image. The CUDA build of PyTorch brings NVIDIA CUDA libraries, which are covered by NVIDIA's license terms. +XGBoost on Linux also installs NVIDIA's NCCL package (`nvidia-nccl-cu12`). diff --git a/cli/src/commands/env.ts b/cli/src/commands/env.ts index fa1aa905..29873dcb 100644 --- a/cli/src/commands/env.ts +++ b/cli/src/commands/env.ts @@ -22,16 +22,16 @@ * WHERE THE ENVIRONMENTS LIE is the main decision of the group, and it is * this. * - * · The ones shipped with the product (base, base-gpu, cv, gpu) are in - * /kernel/environments and are only read: for an installed colloq - * that is site-packages, and it cannot be written to in any sense (see - * env.ts · ownEnvDir). + * · The ones shipped with the product (base, kaggle-base, base-gpu, + * kaggle-gpu) are in /kernel/environments and are only read: for + * an installed colloq that is site-packages, and it cannot be written to + * in any sense (see env.ts · ownEnvDir). * · One's own are in /environments, next to .env and data/. In the * repository that is the very same kernel/environments, and nothing * changes there. * · The list is the union of the two directories; one's own name overrides - * a shipped one, because redefining cv for one's course is a legitimate - * wish. + * a shipped one, because redefining kaggle-base for one's course is a + * legitimate wish. * * Exactly one function turns a name into a path, envFile(); everything else * calls it. A second answer to the question "where does cv lie" would split diff --git a/cli/src/env.ts b/cli/src/env.ts index e32bdf84..ab98a319 100644 --- a/cli/src/env.ts +++ b/cli/src/env.ts @@ -297,12 +297,12 @@ export function createEnv(opts: EnvOptions = {}): Env { * The second directory of environments, and it is created exactly where * the first one cannot be written to. * - * The environments that arrived with the product (base, base-gpu, cv, gpu) - * lie in /kernel/environments. For an installed colloq that is - * site-packages: the directory is overwritten entirely by the next - * `pip install -U`, and on many machines is not writable at all. Creating - * one's own environment there means either a permission refusal or losing - * the file on the very first update. + * The environments that arrived with the product (base, kaggle-base, + * base-gpu, kaggle-gpu) lie in /kernel/environments. For an + * installed colloq that is site-packages: the directory is overwritten + * entirely by the next `pip install -U`, and on many machines is not + * writable at all. Creating one's own environment there means either a + * permission refusal or losing the file on the very first update. * * So one's own lives in the state directory, next to .env and data/, where * nobody will overwrite it: /environments. diff --git a/deploy/helm/colloq/README.md b/deploy/helm/colloq/README.md index b6ed0837..a36a279a 100644 --- a/deploy/helm/colloq/README.md +++ b/deploy/helm/colloq/README.md @@ -48,7 +48,7 @@ namespace fails on those names instead of sharing rooms with the first. | 8 | Secrets from Vault through External Secrets, deployment through ArgoCD. The chart never needs a random value at render time: every secret can come from an existing Secret. | `config.existingSecret`, `secrets.*`, `metrics.existingSecret` | | 9 | Prometheus Operator (optional ServiceMonitor), logs from stdout. | `metrics.*`, `config.logFormat` | | 10 | Backups by volume snapshots (Velero, CSI) plus consistent database snapshots the app writes into its data volume. | `config.dbSnapshotHours`, `config.dbSnapshotKeep` | -| 11 | GPU nodes optional: runtime class, node selector and tolerations of GPU rooms are values. | `gpu.*` | +| 11 | GPU nodes optional: runtime class, node selector and tolerations of GPU rooms are values, and so is the extended resource a GPU competition's submissions request; the device plugin and its sharing (time-slicing, MPS, MIG) are the platform's. | `gpu.*` | ## Storage: ReadWriteOnce or ReadWriteMany @@ -73,6 +73,89 @@ room and personal-notebook Pods mount the workspace, competition Pods the data. `accessMode` to that claim's mode. Claims are annotated to survive `helm uninstall` and the deletion of an ArgoCD application. +## GPU for competition submissions + +Off by default. A teacher may ask for a GPU per competition (on a catalog +environment with `gpu: true`); with `gpu.competitions.resource` set, the +notebook step of each submission then requests `gpu.competitions.count` units of +that extended resource (requests equal limits) and runs with the GPU rooms' +`gpu.runtimeClassName`, `gpu.nodeSelector` and `gpu.tolerations`, and with a +memory-backed `/dev/shm` of half its memory, at most `gpu.competitions.shmMb` +(1024 MiB), for PyTorch's DataLoader workers. The metric step never gets a GPU: +it mounts the hidden answers. Everything else on a GPU environment's image (the metric, package +preparation, a competition that asked for no GPU) runs on the CPU with +`NVIDIA_VISIBLE_DEVICES=void`, so an image built from `nvidia/cuda` gets no card +on a node whose default runtime is NVIDIA's. While the resource is empty the app +refuses submissions to a competition that asks for a GPU and says why; it never +runs them on the CPU instead. + +What one unit is, the platform's NVIDIA device plugin decides (directly, or +through the GPU operator's `devicePlugin.config`), not the chart: + +- **Time-slicing** (`sharing.timeSlicing.resources: [{name: nvidia.com/gpu, replicas: 4}]`): + every card is offered four times and four submissions share it in time, with + no memory or fault isolation between them. With `renameByDefault: true` the + replicas are `nvidia.com/gpu.shared`: set that name here. +- **MPS** (`sharing.mps`, device plugin 0.15 or newer): replicas as above, each + limited to an equal part of the card's memory. **Set `gpu.competitions.shmMb: 0`.** + The plugin mounts its MPS daemon's shared memory at the container's `/dev/shm`, + and the kubelet mounts a Pod's own volumes after the plugin's: a submission's + own `/dev/shm` would hide the daemon's, CUDA could not reach the MPS server, + and every GPU notebook would train on the CPU or fail + ([NVIDIA/k8s-device-plugin#467](https://github.com/NVIDIA/k8s-device-plugin/issues/467)). + With 0 the `/dev/shm` is the daemon's, shared by the node's MPS clients and + sized by it. Rooms keep their own 1Gi `/dev/shm`, so keep them off MPS nodes: + rename the MPS replicas (`renameByDefault: true`, `nvidia.com/gpu.shared`) + and set that name here. +- **MIG** (A100, A30, H100 and newer): slices isolated in hardware. The `mixed` + strategy names them by profile, e.g. `nvidia.com/mig-1g.10gb`; `single` keeps + `nvidia.com/gpu`. + +Rooms always request one `nvidia.com/gpu`. Where the cards a class uses must +never be shared with submissions, give submissions a name of their own (a MIG +profile, or time-sliced replicas renamed to `nvidia.com/gpu.shared` on nodes +labelled for that device-plugin configuration). A count above 1 makes sense for +whole cards and MIG slices; time-sliced replicas of one card add no compute +(`failRequestsGreaterThanOne: true` makes the plugin refuse such requests). + +**ReadWriteOnce storage pins every competition Pod to the app's node** +(`RUNTIME_COLOCATE_WITH_APP=1`, above), GPU submissions included: the units +must be on that node, so the app must run on a GPU node (`app.nodeSelector`, +`app.tolerations`), or the data claim must be ReadWriteMany. Otherwise no node +satisfies both, and every GPU submission waits in the queue "for resources" for +as long as nobody looks. The chart sees no nodes and cannot check this; after +the install, run a competition's baseline check on a GPU environment. + +**How many at once: `gpu.competitions.slots`.** The app's queue keeps that many +GPU submissions at the broker and lets CPU work (other competitions, rescores, +baseline checks) go past the GPU submissions that wait for their turn. Set it +to how many submissions the resource can serve at once, `count` units each, +after what GPU rooms take when they request the same resource. Unset, the queue +sends one at a time. Too high costs throughput, not results: a GPU submission +the scheduler cannot place holds a general slot for 30 seconds and then waits +in the queue again. + +**When Kubernetes refuses a GPU submission's Pod.** A ResourceQuota that names +the resource (`requests.nvidia.com/gpu: "2"`) bounds GPU rooms and GPU +submissions together, and the API server refuses a Pod over it. That refusal, +a missing RuntimeClass (`gpu.runtimeClassName`; `nvidia` is created by the GPU +operator, and absent without it), or an admission policy that turns down the GPU +parts of the Pod sends the submission back to the queue: it is retried a few +seconds to a minute later, and its teacher sees the cause next to it. A CPU job +refused the same way still fails, as it always did. Keep +`gpu.competitions.slots` × `count` within such a quota, less what GPU rooms +hold. + +**Image pull: `gpu.competitions.pullSeconds`.** GPU catalog images are 10 to +13 GB, and every job of a GPU environment runs one: the notebook on a GPU node, +the metric and package preparation on any node of the students' pool. A node +that has never run the image pulls it after the Pod is created, and the broker +waits up to `pullSeconds` (900) for that before the job's own time starts; the +Pod's `activeDeadlineSeconds` grows by as much. A job that started still gets +exactly its own limit. To spare the first class the wait, pull the GPU images +onto those nodes beforehand (for instance, a DaemonSet on those nodes whose init +containers are the GPU images, each running `true`). + ## Secrets, Vault and ArgoCD | Secret | Keys | Mounted by | Replace with | @@ -292,9 +375,14 @@ out so the app's default applies. The app reads the ConfigMap, then the Secret | `rooms.priorityClassName` | `""` | `RUNTIME_PRIORITY_CLASS` (every broker-created Pod) | | `rooms.podLabels` | `{}` | `RUNTIME_POD_LABELS` (JSON, with `app.kubernetes.io/name`, `part-of` and `commonLabels`; never `instance`, which ArgoCD's label tracking would take for its own and prune) | | `rooms.podAnnotations` | `{}` | `RUNTIME_POD_ANNOTATIONS` (JSON) | -| `gpu.runtimeClassName` | `nvidia` | `RUNTIME_GPU_RUNTIME_CLASS` (`""`: none) | -| `gpu.nodeSelector` | `{}` | `RUNTIME_GPU_NODE_SELECTOR` (JSON) | -| `gpu.tolerations` | `[]` | `RUNTIME_GPU_TOLERATIONS` (JSON) | +| `gpu.runtimeClassName` | `nvidia` | `RUNTIME_GPU_RUNTIME_CLASS` (`""`: none); also a competition notebook given a GPU | +| `gpu.nodeSelector` | `{}` | `RUNTIME_GPU_NODE_SELECTOR` (JSON); also a competition notebook given a GPU | +| `gpu.tolerations` | `[]` | `RUNTIME_GPU_TOLERATIONS` (JSON); also a competition notebook given a GPU | +| `gpu.competitions.resource` | `""` (off) | `RUNTIME_COMPETITION_GPU_RESOURCE`, rendered only when set: the extended resource a GPU competition's notebook step requests, e.g. `nvidia.com/gpu`, `nvidia.com/mig-1g.10gb` (see *GPU for competition submissions*) | +| `gpu.competitions.count` | `1` | `RUNTIME_COMPETITION_GPU_COUNT`, with the resource: units per submission, 1..8 | +| `gpu.competitions.slots` | `null` (one at a time) | `RUNTIME_COMPETITION_GPU_SLOTS`, with the resource, rendered only when set: GPU submissions the cluster runs at once, 1..256 | +| `gpu.competitions.shmMb` | `1024` | `RUNTIME_COMPETITION_GPU_SHM_MB`, with the resource: the ceiling of a GPU submission's `/dev/shm`, MiB (also at most half its memory); `0`: none, for MPS | +| `gpu.competitions.pullSeconds` | `900` | `RUNTIME_COMPETITION_GPU_PULL_SECONDS`: the image pull allowance of every Pod of a GPU catalog image, 0..3600 | Set `rooms.maxMemory` on a cluster of unequal nodes: without it the broker takes its own node's memory as every room's ceiling. A room reserves its memory and diff --git a/deploy/helm/colloq/templates/runtime.yaml b/deploy/helm/colloq/templates/runtime.yaml index e95ce2cd..ca014b15 100644 --- a/deploy/helm/colloq/templates/runtime.yaml +++ b/deploy/helm/colloq/templates/runtime.yaml @@ -131,6 +131,24 @@ spec: - name: RUNTIME_GPU_TOLERATIONS value: {{ toJson . | quote }} {{- end }} + {{- /* With the resource or not at all: without one competition jobs get no GPU, and the count, the slots and /dev/shm mean nothing. */}} + {{- with .Values.gpu.competitions.resource }} + - name: RUNTIME_COMPETITION_GPU_RESOURCE + value: {{ . | quote }} + - name: RUNTIME_COMPETITION_GPU_COUNT + value: {{ $.Values.gpu.competitions.count | int | toString | quote }} + {{- /* null is "not said": never an empty variable, which the broker reads as an unparseable number. */}} + {{- with $.Values.gpu.competitions.slots }} + - name: RUNTIME_COMPETITION_GPU_SLOTS + value: {{ . | int | toString | quote }} + {{- end }} + {{- /* 0 is a setting of its own (no /dev/shm of ours, for MPS), so it is passed, not dropped. */}} + - name: RUNTIME_COMPETITION_GPU_SHM_MB + value: {{ $.Values.gpu.competitions.shmMb | int | toString | quote }} + {{- end }} + {{- /* Every Pod of a GPU catalog image, with a GPU or without one. */}} + - name: RUNTIME_COMPETITION_GPU_PULL_SECONDS + value: {{ .Values.gpu.competitions.pullSeconds | int | toString | quote }} {{- with .Values.rooms.priorityClassName }} - name: RUNTIME_PRIORITY_CLASS value: {{ . | quote }} diff --git a/deploy/helm/colloq/templates/validate.yaml b/deploy/helm/colloq/templates/validate.yaml index 01ffb54b..dd55d740 100644 --- a/deploy/helm/colloq/templates/validate.yaml +++ b/deploy/helm/colloq/templates/validate.yaml @@ -76,6 +76,30 @@ behind a green sync. {{- fail "rooms.cpu: at most 64 cores" -}} {{- end -}} +{{- /* A GPU for competition submissions: Kubernetes' rule for an extended resource, as the broker checks it (runtime/src/pod-policy.ts · parseExtendedResource). */ -}} +{{- with $values.gpu.competitions.resource -}} +{{- if or (not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?$" .)) (contains "kubernetes.io/" .) (hasPrefix "requests." .) -}} +{{- fail (printf "gpu.competitions.resource %q: the extended resource your GPU device plugin offers, e.g. nvidia.com/gpu or nvidia.com/mig-1g.10gb" .) -}} +{{- end -}} +{{- $count := int $values.gpu.competitions.count -}} +{{- if or (lt $count 1) (gt $count 8) -}} +{{- fail (printf "gpu.competitions.count %v: 1 to 8 units of %s per submission" $values.gpu.competitions.count .) -}} +{{- end -}} +{{- with $values.gpu.competitions.slots -}} +{{- if or (lt (int .) 1) (gt (int .) 256) -}} +{{- fail (printf "gpu.competitions.slots %v: 1 to 256 GPU submissions at once, or null" .) -}} +{{- end -}} +{{- end -}} +{{- $shm := int $values.gpu.competitions.shmMb -}} +{{- if or (lt $shm 0) (gt $shm 65536) -}} +{{- fail (printf "gpu.competitions.shmMb %v: 0 to 65536 MiB (0: no /dev/shm of ours, for MPS)" $values.gpu.competitions.shmMb) -}} +{{- end -}} +{{- end -}} +{{- $pull := int $values.gpu.competitions.pullSeconds -}} +{{- if or (lt $pull 0) (gt $pull 3600) -}} +{{- fail (printf "gpu.competitions.pullSeconds %v: 0 to 3600 seconds" $values.gpu.competitions.pullSeconds) -}} +{{- end -}} + {{- /* Keys that belong to the chart and the broker. */ -}} {{- range $key, $_ := $values.commonLabels -}} {{- if or (hasPrefix "app.kubernetes.io/" $key) (hasPrefix "colloq.dev/" $key) (hasPrefix "helm.sh/" $key) -}} diff --git a/deploy/helm/colloq/values.schema.json b/deploy/helm/colloq/values.schema.json index 80fe2c6b..37428edb 100644 --- a/deploy/helm/colloq/values.schema.json +++ b/deploy/helm/colloq/values.schema.json @@ -284,7 +284,18 @@ "properties": { "runtimeClassName": { "type": "string" }, "nodeSelector": { "$ref": "#/definitions/stringMap" }, - "tolerations": { "$ref": "#/definitions/objectList" } + "tolerations": { "$ref": "#/definitions/objectList" }, + "competitions": { + "type": "object", + "additionalProperties": false, + "properties": { + "resource": { "type": "string", "pattern": "^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?)?$" }, + "count": { "type": "integer", "minimum": 1, "maximum": 8 }, + "slots": { "type": ["integer", "null"], "minimum": 1, "maximum": 256 }, + "shmMb": { "type": "integer", "minimum": 0, "maximum": 65536 }, + "pullSeconds": { "type": "integer", "minimum": 0, "maximum": 3600 } + } + } } }, "persistence": { diff --git a/deploy/helm/colloq/values.yaml b/deploy/helm/colloq/values.yaml index a2446ce0..4fd80fee 100644 --- a/deploy/helm/colloq/values.yaml +++ b/deploy/helm/colloq/values.yaml @@ -239,8 +239,11 @@ rooms: # COLLOQ_ROOM_NETWORK: rooms reach cluster DNS and nothing else; `none` takes # DNS away too. network: "" - # RUNTIME_ROOM_NODE_SELECTOR / RUNTIME_ROOM_TOLERATIONS: where room and - # personal-notebook Pods run. + # RUNTIME_ROOM_NODE_SELECTOR / RUNTIME_ROOM_TOLERATIONS: where every Pod + # that runs student code goes: rooms, personal notebooks, and competition + # submissions, metrics and package preparation (not its package proxy). A + # GPU submission gets gpu.nodeSelector and gpu.tolerations on top, so its + # node must match both. nodeSelector: {} tolerations: [] # RUNTIME_PRIORITY_CLASS: priorityClassName of every Pod the broker creates. @@ -251,14 +254,44 @@ rooms: podLabels: {} podAnnotations: {} -# Rooms of a catalog environment with gpu: true. +# Rooms of a catalog environment with gpu: true, and the submissions of +# competitions that ask for a GPU. gpu: - # RUNTIME_GPU_RUNTIME_CLASS: the RuntimeClass of GPU rooms, created by your - # platform (the NVIDIA GPU operator makes `nvidia`). Empty: none. + # RUNTIME_GPU_RUNTIME_CLASS: the RuntimeClass of GPU rooms and GPU + # submissions, created by your platform (the NVIDIA GPU operator makes + # `nvidia`). Empty: none. runtimeClassName: nvidia - # RUNTIME_GPU_NODE_SELECTOR / RUNTIME_GPU_TOLERATIONS: added for GPU rooms. + # RUNTIME_GPU_NODE_SELECTOR / RUNTIME_GPU_TOLERATIONS: added for GPU rooms, + # and for a competition notebook given a GPU. nodeSelector: {} tolerations: [] + # A competition whose teacher asked for a GPU gives the notebook step of each + # submission this extended resource (never the metric step, which holds the + # hidden answers); it also gets the runtime class, node selector and + # tolerations above. What one unit is your device plugin decides: a whole + # card, a time-sliced or MPS replica, a MIG slice. Empty: off, and such + # competitions take no submissions. See README, "GPU for competition + # submissions". + competitions: + # RUNTIME_COMPETITION_GPU_RESOURCE, e.g. nvidia.com/gpu or nvidia.com/mig-1g.10gb. + resource: "" + # RUNTIME_COMPETITION_GPU_COUNT: units per submission, 1..8. + count: 1 + # RUNTIME_COMPETITION_GPU_SLOTS: how many GPU submissions the cluster runs + # at once for this release, 1..256 — the units you set aside for + # submissions over `count`. The queue keeps that many at the broker and + # lets CPU work go past the rest. null: one at a time. + slots: null + # RUNTIME_COMPETITION_GPU_SHM_MB: the ceiling of a GPU submission's own + # memory-backed /dev/shm, MiB (it also never exceeds half the job's + # memory). 0: none — required with MPS sharing, whose device plugin mounts + # its own /dev/shm, which ours would hide. + shmMb: 1024 + # RUNTIME_COMPETITION_GPU_PULL_SECONDS: how long any Pod of a GPU catalog + # image (10-13 GB) may spend pulling it before the job's own time starts, + # 0..3600. Applies with or without `resource`: the metric and package + # preparation of a GPU environment run its image on the CPU. + pullSeconds: 900 persistence: # ReadWriteOnce: everything that mounts the data or workspace volume (the app, diff --git a/deploy/k3s/README.md b/deploy/k3s/README.md index c648461a..7e32839c 100644 --- a/deploy/k3s/README.md +++ b/deploy/k3s/README.md @@ -77,8 +77,8 @@ preserved. Kubernetes API access is required from the broker through the cluster network; the app and room Pods do not receive service-account tokens. Environment settings are read as data, never sourced as shell commands. Only the -app variable allowlist becomes a Kubernetes Secret; the two broker memory keys -below go into the broker Deployment. The original configuration +app variable allowlist becomes a Kubernetes Secret; the broker keys below (room +memory, competition GPU) go into the broker Deployment. The original configuration is retained at `/var/lib/colloq/config.env` with mode 0600 for recovery. Runtime auth and room-token derivation secrets are distinct and persistent under `/var/lib/colloq/secrets/`; only the broker receives the latter. @@ -113,6 +113,53 @@ A room that the node cannot fit stays `Pending` with `Insufficient memory` never-scheduled Pod and reports the resource; the teacher sees how much the room asked for and what to lower, students see a short note that the teacher can fix it. +## Competition GPU + +Competition submissions get no GPU unless the env file names one. A +competition on a GPU environment that asks for no GPU runs its notebooks on the +CPU; one that asks for a GPU takes no submissions until this is set: + +```sh +RUNTIME_COMPETITION_GPU_RESOURCE=nvidia.com/gpu +RUNTIME_COMPETITION_GPU_COUNT=1 +``` + +The notebook step of each such submission then requests that many units, with +the `nvidia` RuntimeClass; the metric step never gets one. The installer's +device plugin has no sharing configured, so a unit here is a whole card, taken +from the same cards as GPU rooms for as long as the submission runs; a +submission that finds none free waits in the queue. It works the other way +too: a GPU room that finds every card held by submissions is refused after 20 +seconds (*Room memory* above), for as long as they run, up to the +competition's time limit. On a one-card VM GPU submissions and GPU rooms +therefore exclude each other: leave the resource unset while classes need a +GPU room, or add a card. A resource of their own for submissions (MIG slices, +renamed time-sliced replicas) needs a device-plugin configuration that this +installer does not make. + +Three more settings are optional: + +```sh +RUNTIME_COMPETITION_GPU_SLOTS=1 +RUNTIME_COMPETITION_GPU_SHM_MB=1024 +RUNTIME_COMPETITION_GPU_PULL_SECONDS=900 +``` + +Values are read as written: keep a comment on a line of its own. +`RUNTIME_COMPETITION_GPU_SLOTS` (1 to 256) is how many GPU submissions the +app's queue keeps at the broker at once, while CPU work goes past the rest; +with whole cards and a count of 1, the number of cards submissions may hold. +Unset, it is one, which suits a one-card VM. A GPU submission's `/dev/shm` is +half its memory, at most `RUNTIME_COMPETITION_GPU_SHM_MB` (0 to 65536, 1024 +when unset; 0 gives it none). `RUNTIME_COMPETITION_GPU_PULL_SECONDS` (0 to +3600, 900 when unset) is how long a Pod of a GPU environment's image may take +to pull it before the job's own time starts; the installer pulls every catalog +image onto this node beforehand, so here it is only a reserve. The resource +must be a Kubernetes extended resource name and the numbers whole numbers in +those bounds (the count from 1 to 8); `install`, `prepare` and `update` check +all five settings, render them into the broker Deployment and keep them on +every update, like the room memory above. + ## Storage and recovery Static local PVs use `Retain` and these host paths: @@ -222,7 +269,9 @@ manifest as proof. Containers still share the host Linux kernel. GPU hosts must retain a working NVIDIA driver and install an explicitly pinned NVIDIA Container Toolkit and device-plugin release, configured for k3s/containerd -and the `nvidia` RuntimeClass. Do not run the old Docker-specific runtime setup. +and the `nvidia` RuntimeClass. The driver must be 580 or newer for the shipped +GPU environments: `base-gpu` and `kaggle-gpu` carry torch 2.14.1 as CUDA 13.0 +wheels, which import on an older driver but never see the card. Do not run the old Docker-specific runtime setup. For GPU catalogs, release.tooling.gpu must name an exact toolkitVersion (apt package version) and digest-pinned devicePluginImage. The installer verifies nvidia-smi, installs all four NVIDIA toolkit packages at that exact version @@ -232,6 +281,7 @@ devices, and runs a disposable non-root CUDA tensor Pod on start. Drivers are never replaced and no latest GPU operator is installed. The CUDA probe requires one free GPU; it fails without evicting an existing room if none is available. Use cluster.sh gpu-preflight to repeat it. One GPU room requests one `nvidia.com/gpu`; sharing is not implicit. +A competition notebook given a GPU requests its units from the same cards (*Competition GPU* above). Before promising GPU availability, check node allocatable devices and run an actual CUDA operation in a disposable room. [K3s NVIDIA configuration](https://docs.k3s.io/advanced#nvidia-container-runtime) diff --git a/deploy/vast/README.md b/deploy/vast/README.md index 5cbf9564..b3400f0a 100644 --- a/deploy/vast/README.md +++ b/deploy/vast/README.md @@ -43,7 +43,7 @@ Run as a plain Docker instance, the image refuses to start (exit code 78) and pr ### Other choices -- **No CUDA in the image.** Only room kernels use the GPU. The torch wheels in `kernel/environments/base-gpu.txt` bring their own CUDA runtime, and the driver comes from the host through nvidia-container-toolkit. An `nvidia/cuda` or PyTorch base under the server would add gigabytes to every rental and give the rooms nothing. So there is one image for CPU and GPU machines, and GPU-ness is a property of the **kernel** environment (`# colloq: gpu`). +- **No CUDA in the image.** Only room kernels and GPU submissions use the GPU. The torch wheels in `kernel/environments/base-gpu.txt` bring their own CUDA runtime, and the driver comes from the host through nvidia-container-toolkit. An `nvidia/cuda` or PyTorch base under the server would add gigabytes to every rental and give the rooms nothing. So there is one image for CPU and GPU machines, and GPU-ness is a property of the **kernel** environment (`# colloq: gpu`). - **`NODE_ENV=development`.** The server allows the Docker kernel backend only outside production, because production expects the k3s broker. `deploy/colloq-legacy.service` sets it for the same reason. It changes nothing else: the unsafe file fallback also requires `COLLOQ_UNSAFE_DEV_FILES=1`, and this image never sets that. - **Non-root server.** The entrypoint starts as root only to prepare the state directory and to join the socket's group. The server, tunnels, builds and backups then run as `node` (uid 1000). That is the same uid as the kernel user in `kernel/Dockerfile`, so both sides can write room files without group or umask tricks. Access to the Docker socket is still root on the VM; see [Isolation](#isolation-what-users-get). - **One Colloq per Docker daemon.** The idle sweep removes every container labelled `colloq.kind=room-kernel` on the daemon, not only its own. Do not run a second Colloq on the same VM. `make vast-image-run` tests inside `docker:dind` for exactly this reason. @@ -55,6 +55,8 @@ cp deploy/vast/onstart.sh backups/vast-onstart.sh # once; fill in the settings make vast-vm HOST=class1 # rent the cheapest suitable VM for class1. make vast-vm HOST=class1 GPU="RTX 3070" # only this card make vast-vm HOST=class1 CPU=48 RAM=96 # at least 48 cores and 96 GB, the cheapest card with them +make vast-vm HOST=class1 GPUS=2 GPU_RAM=24 DISK=100 # two cards of 24 GB or more, and disk for the CUDA images +make vast-vm HOST=class1 OFFER= # exactly this offer, wherever it stands in the market make vast-vm INSTANCE= HOST=class2 # redeploy on a VM you already rent ``` @@ -66,11 +68,11 @@ Do not rent from a template in the Vast console. Measured on 28 September 2026: - **Vast does not run the on-start on a VM.** The request carries it, but nothing executes it. `make vast-vm` therefore copies it over SSH to `/etc/colloq/onstart.sh` and starts it. After a reboot nothing runs it again, and nothing needs to: Docker restarts the Colloq container, and the settings live in `/etc/colloq/colloq.env`. - **A running VM reports `actual_status: created`**, so the script waits for SSH, not for a status. -`make vast-vm` reads `VAST_TOKEN`, `VAST_SSH_KEY`, `VAST_IMAGE`, `VAST_DISK`, `VAST_MAX_PRICE` and `VAST_GPU` from `.env`. It picks a verified on-demand VM offer with reliability of at least 0.98 and asks before renting (`FORCE=1` skips the question). For GPU classes it checks the offer's *Max CUDA* (at least 12.1) after the search, not in it: in September 2026 the server-side filter hid RTX 5070 and RTX 5090 offers whose listed value was 13.0. `HOST` overrides `COLLOQ_HOSTNAME` for this machine only, without editing the file. The script then waits for `/api/health` and prints the class address and the owner link. +`make vast-vm` reads `VAST_TOKEN`, `VAST_SSH_KEY`, `VAST_IMAGE`, `VAST_DISK`, `VAST_MAX_PRICE` and `VAST_GPU` from `.env`; `DISK=`, `MAX_PRICE=` and `GPU=` on the command line win for one rental, and `GPUS=` (cards, at least) and `GPU_RAM=` (GB per card, at least) exist on the command line only. It picks a verified on-demand VM offer with reliability of at least 0.98 and asks before renting (`FORCE=1` skips the question). `OFFER=` is asked for by its number (`ask_contract_id`): Vast answers a search with 64 offers at most, so an offer matched only in the answer was found among the cheapest and nowhere else, and an `id` condition returns no offers at all (13 October 2026). The other filters still apply to it. It checks every offer's *Max CUDA* (at least 13.0) after the search, not in it: in September 2026 the server-side filter hid RTX 5070 and RTX 5090 offers whose listed value was 13.0. The floor holds for every rental, a CPU class's too: `base-gpu` and `kaggle-gpu` carry torch 2.14.1 as CUDA 13.0 wheels, which need an NVIDIA driver of 580 or newer, and on an older one PyTorch imports but never sees the card. An offer refused for that alone is reported with the other conditions (`no VM offer matches: …, Max CUDA from 13.0.`). `HOST` overrides `COLLOQ_HOSTNAME` for this machine only, without editing the file. The script then waits for `/api/health` and prints the class address and the owner link. `backups/vast-onstart.sh` is git-ignored because it holds the relay token and API keys. Set at least `COLLOQ_IMAGE=ghcr.io//colloq-vast:` and choose an address (below). `colloq-host` saves the settings to `/etc/colloq/colloq.env` (mode 0600) on the VM. When the on-start runs again, its non-empty values are written over the same keys in that file, and a key it leaves empty keeps the value already there. A copy made before 0.9.0 lacks the two lines under the settings block (`COLLOQ_ON_VAST=1`, `COLLOQ_FREEZE_UPDATES=1`); `colloq-host` still recognises a VM that `make vast-vm` rented by the `/etc/colloq/onstart.sh` it leaves there, but copy the lines over. -A GPU environment needs at least 100 GB of disk (`VAST_DISK=100`): `base-gpu` alone is about 9 GB of torch/CUDA wheels, and each environment adds its own image. 60 GB is enough for CPU classes. +A GPU environment needs at least 100 GB of disk (`DISK=100`, or `VAST_DISK=100` for good): `base-gpu` alone is an image of about 9.9 GB (torch and its CUDA wheels), `kaggle-gpu` adds about 2.8 GB on top of it (12.7 GB in all, measured on 13 October 2026), and each environment adds its own image. 60 GB is enough for CPU classes. After you rent: @@ -102,6 +104,24 @@ With a Cloudflare tunnel the server trusts the `CF-Connecting-IP` header (`TRUST To point a relay name at a machine, set `COLLOQ_HOSTNAME=` together with the `RELAY_*` settings. The relay only serves names under its own zone, and a name that is already taken means another machine is serving that class. The tunnel keeps retrying with a growing pause, so it takes over as soon as the other machine lets the name go. +### GPU rooms and GPU submissions + +```sh +make vast-vm HOST=class1 GPUS=2 GPU_RAM=24 DISK=100 +``` + +In the on-start, `KERNEL_PRELOAD="base,base-gpu,kaggle-gpu"` builds the GPU chain on the first boot: `base-gpu` (CUDA PyTorch and torchvision, about 9.9 GB) and `kaggle-gpu` on top of it (the `kaggle-base` pins plus seaborn, OpenCV and timm, about 2.8 GB more, 12.7 GB in all). Neither image is published, so the first boot takes longer, and the disk holds both next to the data and the backups. Both carry torch 2.14.1 built for CUDA 13.0, so the VM's driver must be 580 or newer; `make vast-vm` rents only offers whose *Max CUDA* is 13.0 or more for that reason. Each image a competition was checked on also keeps a tag of its own (`colloq-revision:`) that nothing removes, so rebuilding `base-gpu` under a running competition leaves the old one, about 10 GB, on disk. + +Every card in `KERNEL_GPUS` serves rooms until the owner gives some to competition submissions in the panel's Resources tab (`COMPETITION_GPUS=1` in the on-start does the same until the panel saves a choice of its own). A card given to submissions leaves the rooms' pool, so a class and a submission never share one; a class already holding it keeps it until its container ends, and a card given back to the rooms serves a class once the submissions running on it end. A submission gets a card only from a competition on a CUDA environment with its GPU switch on, and only for the notebook: the metric never gets one. With no card given, such a competition refuses uploads and says why. + +The second setting is how many submissions share one card at a time (1–8, `COMPETITION_GPU_PER_DEVICE`). It may change during a class: a running submission keeps the share it started with, and a card takes a new one only once its share fits beside the shares already on it. Each gets an equal share of the card's video memory, and the share is enforced, not only promised: + +- inside, `COLLOQ_GPU_MEMORY_MB` (the share less 512 MB for the CUDA context) and a `sitecustomize.py` the harness ships cap PyTorch's allocator, so a model that outgrows the share fails in its cell with `torch.OutOfMemoryError`, also when a library such as polars asked about torch before the notebook imported it. The same file gives a CatBoost GPU fit a `gpu_ram_part` of `COLLOQ_GPU_MEMORY_MB` over `COLLOQ_GPU_CARD_MB` (the card the share was cut from) unless the notebook names its own: CatBoost's own default takes 95 % of the card's free memory at once. XGBoost allocates as it goes and needs nothing; TensorFlow and JAX, in an environment of your own, are not capped, and the watchdog stops them when they take the card; +- outside, the runner reads the container's video memory every 3 seconds as root (`docker exec --user 0:0 nvidia-smi --query-compute-apps=used_memory …`, which inside a container lists that container's processes only, and which the notebook's uid 1000 cannot signal) and kills a submission that is more than 256 MB over its share on two readings in a row. The entrant sees out of memory, with the share and what it took; with a hidden test under the brief output, only the share, since how far past it a run went is a number its notebook chose. One failed reading decides nothing, but the watchdog fails closed: 4 failed in a row (about 12 seconds) stop the run, as a failed notebook if the card had answered in that run (a notebook that starves its own processor does that), and as the server's own failure, with a hint for the teacher, if it never did; +- a burst outside the cap can take a neighbour's memory in the seconds before the second reading, so a submission that runs out of video memory (`torch.OutOfMemoryError`) while the watchdog stops another one on the same card for going past its share goes back to the queue once, and that attempt does not count. + +With one submission per card the whole card is the submission's, and nothing is capped or watched. A card of unknown size (a MIG instance, or any card before a kernel image exists to ask it from) has no share to promise: nothing is capped or watched, and the panel says the size is unknown. Compute is time-shared: three to a card, a model that keeps the card busy trains about three times slower, inside the same time limit. A small one leaves the card idle between its steps and hardly notices: the small CNN of the load test below ran in 29 seconds eight to a card, against 25 seconds four to a card. The share protects memory, not speed. + ## Settings The server reads the container's environment and `/workspace/colloq/.env`. For most settings the container environment wins, because `.env` only fills variables that are not already set. `PUBLIC_URL` and `KERNEL_ENV` work the other way: the server re-reads them from `.env` while it runs, and the file wins. That lets the tunnel and the panel's "Make default" change them without a restart. The entrypoint writes `PUBLIC_URL` into `.env` on every start. @@ -119,7 +139,7 @@ The server reads the container's environment and `/workspace/colloq/.env`. For m | `INSTITUTION` | — | Shown next to the logo | | `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL`, `AI_PROVIDER`, `AI_REASONING` | off, OpenAI, `gpt-4o-mini`, `openai`, `false` | The assistant. It can also be configured later in the panel | | `KERNEL_ENV` | `base` | Default environment for new rooms (the panel's "Make default" writes it to `.env`) | -| `KERNEL_PRELOAD` | `KERNEL_ENV` | Environments to prepare at start, comma-separated, e.g. `base,base-gpu` or `base,kaggle-base` | +| `KERNEL_PRELOAD` | `KERNEL_ENV` | Environments to prepare at start, comma-separated, e.g. `base,kaggle-base`, `base,base-gpu` or `base,base-gpu,kaggle-gpu` | | `KERNEL_IMAGE_REPO` | the published `ghcr.io/colloq-edu/colloq-kernel` | Where prebuilt kernel images come from, as `:-`. By default the published ones are pulled for the shipped, unedited `base` and `kaggle-base` of this version, and everything else is built on the machine. An explicit repository is asked for every environment; `none` builds them all on the machine | | `KERNEL_IMAGE_TAG` | `v` | Tag prefix, matching the names `publish.yml` and `scripts/release-build.py` publish | | `KERNEL_MEM`, `KERNEL_MEM_` | `4g`, `16g` for GPU environments | Memory per room: `KERNEL_MEM` for ordinary environments, `KERNEL_MEM_` for one environment, where `` is its name in capitals with `_` for `-` (`KERNEL_MEM_BASE_GPU`, `KERNEL_MEM_KAGGLE_BASE`). The panel can override it per room, and the owner for the instance (`/api/admin/resources`) | @@ -127,7 +147,9 @@ The server reads the container's environment and `/workspace/colloq/.env`. For m | `KERNEL_OWN_MAX`, `KERNEL_OWN_PIDS` | `60`, `2048` | The second container a class gets for its students' personal notebooks, which never receives a GPU: how many kernels may live in it at once, and its process ceiling | | `KERNEL_OWN_IDLE_MIN` | `30` | Minutes of idling after which one personal notebook's kernel is stopped; the container goes with the last kernel in it. `0` never stops them | | `KERNEL_ROOM_IDLE_MIN` | `120` | Minutes a class with nobody inside and nothing computing keeps its containers before they are stopped and their memory is free again. `0` keeps them. The owner can also stop kernels by hand on the panel's Resources tab | -| `KERNEL_GPUS` | all GPUs (detected by `colloq-host`) | GPU slices handed out one per GPU room, e.g. `0,1` or MIG ids | +| `KERNEL_GPUS` | all GPUs (detected by `colloq-host`) | GPU slices handed out one per GPU room, e.g. `0,1` or MIG ids; the ones given to competition submissions are kept from rooms | +| `COMPETITION_SLOTS` | `auto` | Submissions running at once: `auto` (sized by the machine) or 1–64. The panel's Resources tab wins over it | +| `COMPETITION_GPUS`, `COMPETITION_GPU_PER_DEVICE` | none, `1` | The `KERNEL_GPUS` entries given to competition submissions, comma-separated, and how many submissions share one card at a time (1–8); see [GPU rooms and GPU submissions](#gpu-rooms-and-gpu-submissions). The panel's Resources tab wins over them | | `KERNEL_SHM` | `1g` | `/dev/shm` for GPU rooms | | `KERNEL_NETWORK` | `colloq` | Docker network shared by the server and room kernels | | `MAX_UPLOAD_MB`, `MAX_SESSION_MB`, `COUNCIL_COPY_MB`, `COUNCIL_MEMORY_GUARD`, `OPEN_SEMINAR_CREATION`, `TZ` | `50`, `1024`, `512`, `1`, `false`, — | As in `.env.example` | @@ -239,7 +261,7 @@ Before you make the package public, read [THIRD_PARTY_NOTICES.md](../../THIRD_PA The image carries OCI labels (`org.opencontainers.image.version`, `.revision`, `.source`, `.licenses`), and `colloq-vast version` prints the version and revision it was built from. The frpc 0.71.0 and cloudflared 2026.9.1 downloads are pinned by sha256 in the Dockerfile. When you bump one of them, update the version and both checksums together, and keep frpc on the relay's version. -Prebuilt kernel images make the first boot fast, and an offline install possible. Without them, the default `base` environment builds in a few minutes, and `base-gpu` downloads about 3 GB of wheels. The `kernels` job of `publish.yml`, under the same `PUBLISH_IMAGES` switch, publishes `ghcr.io//colloq-kernel:vX.Y.Z-base` and `:vX.Y.Z-kaggle-base` for linux/amd64, with the build the entrypoint and the panel run, after a check inside each image (the `kernel/locks` check script for `kaggle-base`). The app image is built with that repository as its default (`COLLOQ_KERNEL_REPO`), so the entrypoint pulls them without any setting. `base-gpu` is not published: it is about 9 GB, and a GPU machine builds it once. `scripts/release-build.py --registry ghcr.io//colloq` (the manual k3s workflow, `.github/workflows/release.yml`) uses the same names for the environments listed there. Make the new `colloq-kernel` and `colloq-server` packages public once, as `colloq-vast` was. +Prebuilt kernel images make the first boot fast, and an offline install possible. Without them, the default `base` environment builds in a few minutes, and `base-gpu` downloads about 3 GB of wheels. The `kernels` job of `publish.yml`, under the same `PUBLISH_IMAGES` switch, publishes `ghcr.io//colloq-kernel:vX.Y.Z-base` and `:vX.Y.Z-kaggle-base` for linux/amd64, with the build the entrypoint and the panel run, after a check inside each image (the `kernel/locks` check script for `kaggle-base`). The app image is built with that repository as its default (`COLLOQ_KERNEL_REPO`), so the entrypoint pulls them without any setting. `base-gpu` and `kaggle-gpu` are not published: `base-gpu` is about 9.9 GB, the job refuses any GPU environment (and `kaggle-gpu` through its parent), and a GPU machine builds the chain once. `scripts/release-build.py --registry ghcr.io//colloq` (the manual k3s workflow, `.github/workflows/release.yml`) uses the same names for the environments listed there. Make the new `colloq-kernel` and `colloq-server` packages public once, as `colloq-vast` was. ## Local smoke test @@ -290,13 +312,32 @@ These checks ran on 18 September 2026 on an arm64 laptop, using `docker:dind` as On a real Vast VM (28 September 2026, RTX 3070, `make vast-vm`): the image pulled from GHCR, `colloq-host` found the GPU and started Colloq, the default environment built, and the class address answered through the relay. The same run found the console, on-start and status behaviour described under [Renting a machine](#renting-a-machine-make-vast-vm). It also found that unattended-upgrades upgraded the NVIDIA userspace under the loaded kernel module 20 minutes after boot ("Driver/library version mismatch"). On a Vast VM `colloq-host up` therefore turns automatic updates off and holds the NVIDIA packages on every start (`COLLOQ_FREEZE_UPDATES=1`, which `onstart.sh` sets; a VM started from an older copy of the on-start is recognised by `/etc/colloq/onstart.sh`, which `make vast-vm` leaves there). +On a rented VM on 13 October 2026 (61 vCPU, 98 GB, 2 × RTX 4090 24 GB, driver 580.95.05, CUDA 13.0, colloq 0.21.0): a competition on `base-gpu` ran CUDA PyTorch without a card (`torch.cuda.is_available()` false, no `nvidia-smi`) and said nothing about it, which is what [GPU submissions](#gpu-rooms-and-gpu-submissions) replace. The pieces they are built from were tried there by hand: with `--gpus device=0` a submission container's hardening (read-only root, every capability dropped, uid 1000, no network) still held, and PyTorch saw the card; two containers on one card each saw only their own processes in `nvidia-smi`; a 4096 MB cap through `sitecustomize.py` stopped PyTorch at the 17th 256 MiB allocation with `OutOfMemoryError`; the CUDA context took about 458 MB of video memory. + +Later that day the same VM ran the build that adds GPU submissions, an unreleased image built there from source and installed with `colloq-host update`, with a `kaggle-gpu` built by hand at the versions `kernel/environments/kaggle-gpu.txt` now pins: + +- **The share, inside.** Four submissions to a card gave each a share of 6141 MB, and PyTorch's allocator in a submission was capped at 5.50 GiB, the share less 512 MB. A model that outgrew it failed in its cell with `torch.OutOfMemoryError`, followed by our line naming the share and advising a smaller batch or model. +- **The share, outside.** A submission that took 10.4 GB of the card, past its share, was stopped by the watchdog and ended as out of memory, with the share in the message. +- **Rooms next to submissions.** With both cards given to submissions, a `kaggle-gpu` class was refused with the words for that case. With card 1 given back to the rooms, the class opened on it, and PyTorch in the room saw that RTX 4090 (`NVIDIA_VISIBLE_DEVICES=1`) while submissions ran on card 0. +- **The panel and the page.** The Resources tab listed both cards with their names and memory, asked through Docker from inside the app's container (that build ran one probe container per card; the single container for every card described below was run there by hand: about 6 seconds for both cards, against about 7 for each card alone), and the entrant's page showed the share (≈ 6 GB, `cuda:0`). +- **A class at once.** 25 entrants sent the reference solution of a letter-recognition competition (a small CNN; 2 cores, 4 GB and 600 seconds per submission; `kaggle-gpu`) at the same moment, and every submission scored. A run is the time a submission spent running; a verdict counts from its upload. A mixed load of 10 GPU and 4 CPU submissions kept the queue's rule: the CPU ones did not wait behind the GPU ones. + +| Cards for submissions | Run, median | Verdict, median | Verdict, last | +|---|---|---|---| +| none: CPU only, on 0.21.0 (22 runners × 2 cores) | 159 s | 162 s | 235 s | +| 1 card × 4 | 25 s | 102 s | 175 s | +| 2 cards × 4 | 29 s | 58 s | 88 s | +| 1 card × 8 (a share of about 3 GB) | 29 s | 60 s | 117 s | +| 2 cards × 8 | 31 s | 36 s | 62 s | + Not verified yet: -- GPU rooms with nvidia-container-toolkit on a rented card; +- `kaggle-gpu` built by `KERNEL_PRELOAD` on a first boot: the image on that VM was built by hand, at the same versions; +- on a card, what changed after that run's review: the watchdog reading as root inside the hardened container and stopping a run whose card cannot be read, the cap after a library that asks about torch first, CatBoost's `gpu_ram_part` default, and the retry after a neighbour's burst (each checked with stand-ins only); - pulling the kernel images `publish.yml` publishes from 0.9.0; - on a real Linux VM rather than with stand-ins: `colloq-host doctor`, `bundle` and `load` on an offline machine, `backup-timer`, the backup and rollback around `update`, and the trusted-proxy default behind a proxy on the same host. -Known gap: the panel's *Resources* section does not list the GPU cards. The server reads them with `nvidia-smi`, which exists on the VM but not inside the `colloq-vast` container. GPU rooms still get their slices, because those come from `KERNEL_GPUS`, which `colloq-host` fills in from the host's `nvidia-smi`. +The panel's *Resources* tab used to list no cards on this image: `nvidia-smi` exists on the VM but not inside the `colloq-vast` container. Now, for the `KERNEL_GPUS` entries that no local `nvidia-smi` answers for, the server asks the cards themselves through Docker: one throwaway container of a kernel image already on the machine, given every card (`docker run --rm --gpus all --network none --entrypoint nvidia-smi --query-gpu=index,uuid,name,memory.total …`), where the NVIDIA toolkit mounts `nvidia-smi`; an entry that container does not name is asked from a container given only that card (`--gpus device=`). The cards are asked when the server starts, and the answer is cached for ten minutes and refreshed in the background; a size once read is kept when a later refresh gets no answer. Until a kernel image exists, or for a MIG instance (the query would describe the whole physical card), the size stays unknown and the panel says so. GPU rooms and submissions get their cards from `KERNEL_GPUS` either way, which `colloq-host` fills in from the host's `nvidia-smi`. ## Relation to the other Vast paths diff --git a/deploy/vast/colloq-host b/deploy/vast/colloq-host index e8272c2d..1e74503a 100755 --- a/deploy/vast/colloq-host +++ b/deploy/vast/colloq-host @@ -92,6 +92,7 @@ CLOUDFLARE_TUNNEL_TOKEN OPENAI_API_KEY OPENAI_BASE_URL OPENAI_MODEL AI_PROVIDER ADMIN_EMAIL INSTITUTION UI_LANGUAGE TZ SESSION_SECRET OPEN_SEMINAR_CREATION MAX_UPLOAD_MB MAX_SESSION_MB COUNCIL_COPY_MB COUNCIL_MEMORY_GUARD KERNEL_ENV KERNEL_PRELOAD KERNEL_IMAGE_REPO KERNEL_IMAGE_TAG KERNEL_MEM KERNEL_CPUS KERNEL_GPUS KERNEL_SHM KERNEL_OWN_MAX KERNEL_OWN_PIDS KERNEL_OWN_IDLE_MIN KERNEL_ROOM_IDLE_MIN PUBLIC_IPADDR VAST_TCP_PORT_3000 +COMPETITION_SLOTS COMPETITION_GPUS COMPETITION_GPU_PER_DEVICE TRUSTED_PROXIES TRUST_CF_CONNECTING_IP SHARED_ADDRESSES HSTS HTTPS_PROXY HTTP_PROXY NO_PROXY https_proxy http_proxy no_proxy NODE_EXTRA_CA_CERTS DEPENDENCY_INDEX_URL DEPENDENCY_FILES_HOSTS COLLOQ_ROOM_NETWORK KERNEL_BLOCKED_CIDRS COLLOQ_HELPER_IMAGE diff --git a/deploy/vast/onstart.sh b/deploy/vast/onstart.sh index 12319051..ffc244b3 100755 --- a/deploy/vast/onstart.sh +++ b/deploy/vast/onstart.sh @@ -27,7 +27,15 @@ export ADMIN_EMAIL="" # prefills the owner claim form export INSTITUTION="" # shown next to the logo export OPENAI_API_KEY="" # empty: the assistant is off (can be set later in the panel) -# Python environments prepared before the first class. On a GPU offer: base,base-gpu +# Python environments prepared before the first class. On a GPU offer: +# base,base-gpu GPU rooms (CUDA PyTorch) +# base,base-gpu,kaggle-gpu and a GPU competition (the kaggle-base stack on CUDA) +# The GPU ones are built here on the first boot: base-gpu is about 9.9 GB and +# kaggle-gpu adds about 2.8 GB (12.7 GB in all), so rent such a VM with 100 GB +# of disk or more (make vast-vm DISK=100). Submissions get a card only when +# the owner gives them one in the panel (Resources > GPUs for submissions), or +# with COMPETITION_GPUS="1" COMPETITION_GPU_PER_DEVICE="2" exported here: +# card 1 for submissions, two at a time; the other cards stay with the rooms. export KERNEL_PRELOAD="base" # Where prebuilt kernel images come from, as :v-. Empty: # the published ghcr.io/colloq-edu/colloq-kernel ones (base, kaggle-base), and diff --git a/docker-compose.yml b/docker-compose.yml index 43bafd55..7493d9e4 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -50,6 +50,9 @@ services: KERNEL_MEM_BASE_GPU: ${KERNEL_MEM_BASE_GPU:-} KERNEL_CPUS: ${KERNEL_CPUS:-2} KERNEL_GPUS: ${KERNEL_GPUS:-} + # Cards of KERNEL_GPUS given to competition submissions, and runs per card. + COMPETITION_GPUS: ${COMPETITION_GPUS:-} + COMPETITION_GPU_PER_DEVICE: ${COMPETITION_GPU_PER_DEVICE:-} KERNEL_SHM: ${KERNEL_SHM:-1g} # The second container per class, the one holding students' personal # notebooks: how many kernels may live in it and how many processes it diff --git a/docs/deployment-vast.md b/docs/deployment-vast.md index 75ee8813..20cc4565 100644 --- a/docs/deployment-vast.md +++ b/docs/deployment-vast.md @@ -32,7 +32,7 @@ Existing service/Compose installations, unmanaged nonempty data, and old `.db`/f ## GPU readiness -One GPU request allocates one GPU by default; simultaneous GPU rooms require enough devices or an explicit sharing policy. There is no implicit sharing or fallback to a shared Jupyter endpoint. GPU catalogs require `tooling.gpu.toolkitVersion` (an exact NVIDIA apt version) and `tooling.gpu.devicePluginImage` (a registry digest). The installer preserves the guest driver, installs the pinned toolkit/plugin, checks NVIDIA RuntimeClass and allocatable devices, and runs a disposable PyTorch CUDA tensor operation. `vast-up` also invokes `cluster.sh smoke` for independent room execution before reporting deployment success. Application HTTP health alone does not prove GPU execution works. [K3s NVIDIA support](https://docs.k3s.io/advanced#nvidia-container-runtime), [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) +One GPU request allocates one GPU by default; simultaneous GPU rooms require enough devices or an explicit sharing policy. There is no implicit sharing or fallback to a shared Jupyter endpoint. GPU catalogs require `tooling.gpu.toolkitVersion` (an exact NVIDIA apt version) and `tooling.gpu.devicePluginImage` (a registry digest). The installer preserves the guest driver, which must be 580 or newer for the shipped GPU environments (their torch 2.14.1 is built for CUDA 13.0, and `vast-up` takes only offers whose Max CUDA is 13.0 or more), installs the pinned toolkit/plugin, checks NVIDIA RuntimeClass and allocatable devices, and runs a disposable PyTorch CUDA tensor operation. `vast-up` also invokes `cluster.sh smoke` for independent room execution before reporting deployment success. Application HTTP health alone does not prove GPU execution works. [K3s NVIDIA support](https://docs.k3s.io/advanced#nvidia-container-runtime), [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) Nested KVM and GPU compatibility of stronger optional sandboxes are not assumed. No paid Vast rental or live GPU test is performed by local repository tests. diff --git a/docs/pages/en/competitions.html b/docs/pages/en/competitions.html index e6be6fbc..6f039e0c 100644 --- a/docs/pages/en/competitions.html +++ b/docs/pages/en/competitions.html @@ -24,12 +24,14 @@

The public and the private share

The metric is computed tw

The hidden test

An open data/test.csv is downloaded by the class, and an answer is easy to fit to it: count the rows, copy the ids. A hidden test is a file the entrant does not download: at every check the notebook reads it at the same path, in place of the open example. The model is checked on rows nobody has seen.

  1. In the editor's Hidden test section, upload the file through its own button — not through the open data. In Path during the check give its name in data/; by default it is the file's own name. A file named like an open example replaces the example; a file with a new name is added to the data.
  2. The example stays with the entrants: they download it and debug their notebook on it. Check the columns — the editor shows whether they match the example. A notebook written for the example fails on a test with other columns.
  3. The answers in solution.csv must belong to the hidden test's rows.
  4. Run the sample notebook's check again: a change to the hidden test makes its last check stale.

Every run gets a data/ directory of its own: the open files, with the hidden ones in their place. data/test.csv and /data/test.csv both read the hidden test; the metric never sees it. A file replaced in the middle of a run reaches the next runs only. The ceiling is 200 MB and 10 files, apart from the open data.

Entrants see the hidden file's name and row count: the example carries an EXAMPLE mark in the data list, and the check conditions have a line about the hidden test. No link hands out its contents — not to the entrant and not to you: the hidden test cannot be downloaded.

While the notebook runs, the hidden rows are in its memory, and any output can carry them out: test.head(), a KeyError message with a value in it, a metric message quoting the answer. That is why What participants see of a check has two options:

  • Only the outcome and the error type — the default. The entrant sees the outcome, the number of the failed cell and the exception type from a short list of common ones (KeyError, ValueError…). No cell output, no error text, no traceback; the notebook link hands out the notebook as it was sent, not the executed one. A metric refusal is retold in our own words, without values, ids or counts from the answer.
  • Full output — as without a hidden test. Turn it on only if revealing those rows does no harm: fitting the answer to them becomes possible.

In the feed you always see the whole output and the executed notebook. The full text is kept under either option, so a switch applies to every submission at once, with nothing run again.

-

How a submission is executed

The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in.

Entrants download the open files one by one, or with Download all above the list: the archive holds data/ with every file and the starter notebooks next to it, as in the check, so pd.read_csv('data/train.csv') works right after unpacking. Hidden answers and the hidden test are never in it.

  1. The submission joins the instance queue.
  2. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv.
  3. That file is taken out of the container and the container is destroyed.
  4. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here.

The competition's limits (notebook time, memory, cores) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full. Each submission has a "Download the notebook with its output" link: the notebook as the server executed it — a failed one too, up to the cell where it stopped. Its output is this run's alone: whatever came from the laptop is cleared before the run.

OutcomeWhat the entrant sees
DoneThe public score and their place.
Notebook failed"Failed on cell 5 of 14" and their traceback verbatim.
Time limit"Did not fit into 10 minutes, cell 6 of 50".
Out of memory"Took more than 4 GB, cell 2".
Answer not accepted"Reached the end but left no submission.csv", "The answer is missing 3 rows, for example id=17" — or the text of your ParticipantVisibleError.
Metric failed"The checking code failed", with no detail; you get the traceback and the submission waits to be rescored.

That is a competition without a hidden test. With one, an entrant by default sees a failed submission briefly: the outcome, the cell number and the exception type, with no output and no traceback; see the hidden test.

+

How a submission is executed

The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in.

Entrants download the open files one by one, or with Download all above the list: the archive holds data/ with every file and the starter notebooks next to it, as in the check, so pd.read_csv('data/train.csv') works right after unpacking. Hidden answers and the hidden test are never in it.

  1. The submission joins the instance queue.
  2. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv.
  3. That file is taken out of the container and the container is destroyed.
  4. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here.

The competition's limits (notebook time, memory, cores, and the GPU switch) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full. Each submission has a "Download the notebook with its output" link: the notebook as the server executed it — a failed one too, up to the cell where it stopped. Its output is this run's alone: whatever came from the laptop is cleared before the run.

OutcomeWhat the entrant sees
DoneThe public score and their place.
Notebook failed"Failed on cell 5 of 14" and their traceback verbatim.
Time limit"Did not fit into 10 minutes, cell 6 of 50".
Out of memory"Took more than 4 GB, cell 2".
Answer not accepted"Reached the end but left no submission.csv", "The answer is missing 3 rows, for example id=17" — or the text of your ParticipantVisibleError.
Metric failed"The checking code failed", with no detail; you get the traceback and the submission waits to be rescored.

That is a competition without a hidden test. With one, an entrant by default sees a failed submission briefly: the outcome, the cell number and the exception type, with no output and no traceback; see the hidden test.

-

The queue and the console

There is one queue for the whole instance; it lives in the database and survives a server restart. How many submissions run at once is worked out from the machine by default (Auto): its cores and memory, less 2 cores and 2 GB kept for the class, divided by a new submission's default cores and memory. The owner can set a fixed number in the panel's Resources tab; see sizing. The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session.

The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue, and a run in progress has a Kill button.

To stop anyone from fitting the public leaderboard with dozens of submissions, set a daily submission quota. Only submissions that get a score spend it. Failed ones — a notebook error, the time or memory limit, a rejected answer, a failure of the metric or of the server itself — and cancelled ones do not; a queued submission holds a place for now and gives it back if it fails. A separate ceiling on failed submissions stops endless failures: at most 10 a day per entrant, or three times the quota if that is more (metric failures do not count toward it). An entrant who reaches it gets a refusal of its own, apart from the quota. With no quota (0) there is no ceiling either.

+

A card for submissions

A notebook that trains a network wants a video card. A submission gets a share of one when two things are true: the server owner has given cards to submissions, and the competition asks for one.

  1. The owner gives cards. In the panel's Resources tab, the GPUs for submissions block lists the machine's cards (KERNEL_GPUS) with their names and memory. Tick the ones that serve submissions and say how many submissions share one card at a time, from 1 to 8. A card given to submissions leaves the rooms' pool, so a class and a submission never share a card; a class that holds the card at that moment keeps it until its container ends, and submissions wait for it until then; the other way round, a card given back to the rooms serves a class once the submissions running on it end. On a machine set up without the panel, COMPETITION_GPUS (entries of KERNEL_GPUS, comma-separated) and COMPETITION_GPU_PER_DEVICE do the same; what the panel saved wins.
  2. The competition asks. In the editor, pick an environment with CUDA (base-gpu, kaggle-gpu or one of your own on top of them) and turn the GPU switch on. Only such environments have it, and while it is on the environment cannot be changed to one without CUDA. Only the notebook gets the card. The metric never does: it holds the hidden answers and has nothing to train.

The share. Submissions on one card get equal parts of its memory: on a 24 GB card with 3 at a time, 8 GB each. With cards of different sizes, the share promised is the smallest card's: it holds on whichever card a run lands. The number per card may change during a class: a running submission keeps the share it started with, and a card takes a new one only once its share fits beside theirs, so a card raised from 1 to 2 while a submission has it whole takes nobody else until that one ends.

How the share is held. From two sides. Inside the submission, PyTorch is capped at the share less 512 MB for the CUDA context, also when a library such as polars looks for it before the notebook imports it: a model that outgrows the cap fails in its cell with torch.OutOfMemoryError, as it would on a smaller card. CatBoost on the GPU takes 95 % of the card's free memory at once by default; within a share its gpu_ram_part becomes the share's part of the card less the CUDA context (about 0.48 with 2 to a 24 GB card), unless the notebook sets its own. XGBoost on device="cuda" takes memory as it goes, not up front. TensorFlow and JAX, in an environment of your own, are not capped: by default both take most of the card at once, and the runner stops them for it, unless the notebook holds them back before importing them (TF_FORCE_GPU_ALLOW_GROWTH=true, XLA_PYTHON_CLIENT_PREALLOCATE=false). From outside, every 3 seconds the runner reads how much of the card the submission's container really holds, PyTorch or not, and stops a submission that is more than 256 MB over its share twice in a row. It reads as root, which the notebook's own code cannot signal, and it fails closed: a submission whose card cannot be read 4 times in a row (about 12 seconds) is stopped too, because a run nobody watches may be holding the whole card. With 1 submission per card, the whole card is the submission's, and nothing is capped or watched. A card that cannot tell its memory has no share to promise: the panel says its size is unknown, and nothing is capped or watched either.

What the entrant sees. The competition page shows the share of the card next to the other limits. A notebook that hit the cap inside ends as Notebook failed with that torch.OutOfMemoryError and a line of ours naming its share; a submission stopped from outside ends as Out of memory, naming its share and how much it took. With a hidden test, under the default Only the outcome and the error type, the message names only the share: how far past it a run went is a number its notebook chose (see the hidden test). A submission whose card stopped answering is stopped as Notebook failed and counts as a failed one: that happens when a notebook keeps its processor busy with hundreds of threads or processes, or when the card itself stops answering. If the card never answered during the run, the host is to blame, not the notebook: the submission ends as the server's own failure, which counts toward nothing, and the teacher's log says to check nvidia-smi inside a container. A burst outside the cap can take a neighbour's memory in the seconds before the runner's second reading: a submission that runs out of video memory (torch.OutOfMemoryError) while the runner is stopping another one on the same card for going past its share goes back to the queue once, with a line saying so, and that attempt does not count. If no card is given to submissions, a competition with GPU on refuses uploads and the baseline check outright and says why: nothing waits in the queue for a card that will not come.

What a share does not give. Submissions on one card take turns at its compute: three at a time, a model that keeps the card busy trains about three times slower than alone, and the notebook time limit counts that time. The share protects memory, not speed: raise the time limit, or put fewer submissions on a card. A small model leaves the card idle between its steps and hardly notices its neighbours; see the measurement below. In the queue a GPU submission needs a free runner and a free share of a card; while it waits for the card, CPU submissions behind it go ahead.

What a card gives a class. Measured on 13 October 2026 on a Vast VM with 61 vCPU, 98 GB of memory and 2 × RTX 4090 (24 GB each): 25 entrants sent the reference solution of a letter-recognition competition — a small convolutional network; 2 cores, 4 GB and 600 seconds per submission; kaggle-gpu — at the same moment, and every submission scored. A run is the time a submission spent running; a verdict counts from its upload.

Cards for submissionsRun, medianVerdict, medianVerdict, last
none: CPU only, on 0.21.0 (22 runners × 2 cores)159 s162 s235 s
1 card × 425 s102 s175 s
2 cards × 429 s58 s88 s
1 card × 8 (a share of about 3 GB)29 s60 s117 s
2 cards × 831 s36 s62 s

A card makes one run of this notebook five to six times shorter, and after that the class waits for the queue, not for the run: eight to a card ran nearly as fast as four, and two cards with eight each gave all 25 their verdict in about a minute. A bigger model gains less from more submissions to a card, and must fit into the smaller share. A mixed load of 10 GPU and 4 CPU submissions on the same machine kept the queue's rule: the CPU ones did not wait behind the GPU ones.

On Kubernetes the cluster decides what a share is: the chart value gpu.competitions.resource names the device-plugin resource a GPU submission requests, a whole card, a time-sliced replica or a MIG slice; see operations. How many GPU submissions go to the cluster at once is gpu.competitions.slots, one while it is unset: the queue keeps that many there and lets CPU submissions go past the rest, as past a full card here. Colloq caps and watches nothing there itself: a MIG slice has memory of its own, while time-sliced replicas share the card's memory with no limit between them. Nor does a card leave the rooms there: GPU rooms request nvidia.com/gpu, so with that name submissions take their units from the rooms' cards, and a GPU room that finds none free is refused while submissions hold them. Where classes use GPU rooms, give submissions a resource of their own; see GPU nodes.

+ +

The queue and the console

There is one queue for the whole instance; it lives in the database and survives a server restart. How many submissions run at once is worked out from the machine by default (Auto): its cores and memory, less 2 cores and 2 GB kept for the class, divided by a new submission's default cores and memory. The owner can set a fixed number in the panel's Resources tab; see sizing. The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session.

The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue, and a run in progress has a Kill button. An owner deletes a competition with Delete competition in its row's menu in the list, and its runs go first: the waiting submissions leave the queue, and the running ones are killed and waited for, up to 30 seconds each. If one has not stopped by then, nothing is deleted, though the waiting ones stay cancelled, and the delete can be repeated a minute later.

To stop anyone from fitting the public leaderboard with dozens of submissions, set a daily submission quota. Only submissions that get a score spend it. Failed ones — a notebook error, the time or memory limit, a rejected answer, a failure of the metric or of the server itself — and cancelled ones do not; a queued submission holds a place for now and gives it back if it fails. A separate ceiling on failed submissions stops endless failures: at most 10 a day per entrant, or three times the quota if that is more (metric failures do not count toward it). An entrant who reaches it gets a refusal of its own, apart from the quota. With no quota (0) there is no ceiling either.

Entrants and the sign-in key

An entrant is not a room member: their identity belongs to the whole instance. On joining, a person gives their Telegram username or email — the name they appear under on the leaderboard — and receives a sign-in key shaped K7Q-M2X-9FD and a link carrying it. The same key brings them back to their submissions from another device, another browser, or after clearing site data. A username or an address is one person: joining under one the competition already has is refused with a pointer to the sign-in key, and no second identity with another key is created.

The key sits in the "YOUR SIGN-IN KEY" card on the competitions page — shown every time, not once. The key itself is not in the database: what is stored is its fingerprint and a copy encrypted with the instance secret, so a stolen database without that secret opens no one's account.

If an entrant loses their key, issue a new one from the Participants tab. The old one stops working that same second, together with every tab still open on it.

The deadline and the debrief

When the deadline passes, submissions close by themselves and the final leaderboard opens — if the settings say "open it by itself after the deadline". The other option is I will open it by hand — at the review: then a button reveals the results and you choose the minute of the class when they appear.

Finish now closes submissions early. After the deadline there is no need to press it: the competition counts as finished anyway — participants and the panel list it among the finished ones, and the console header offers Close late intake or Take late submissions instead of Finish now, plus Open the private leaderboard when the results open by hand.

For those finishing their solution after the bell there is the Late submissions switch in Dates and standings. With it on, submissions keep being accepted after the deadline and after Finish now: they run, get a public score and spend the daily quota by the same rules as ordinary ones. But a late submission never counts: it is in no places, no leaderboard, no summary and not in the wait for the final results, and it cannot be picked to count. The entrant sees its final score when the results open. On their page it is marked “after the deadline · not counted”, and in the console it carries a LATE mark and a Late filter of its own. A submission is late when its upload began after the deadline; one begun on time and finished within two minutes stays in the standings. A late submission never takes the place of an on-time one still waiting for its check. While late intake is open, the lists say so: participants see the competition among the finished ones marked “late submissions open”, with a button, and the panel writes “late ones accepted” under FINISHED, with how many came.

If you move the deadline later on a running competition, submissions sent before the new deadline become ordinary ones: they join the leaderboard and the standings. Moving it earlier makes no one late. After Finish now moving it changes nothing: the standings were closed by hand.

For the debrief there is Leaderboard on the projector — a separate window with just the table, without file names or internal marks.

-

Additional packages

The Packages tab shows the pinned Python version and complete base inventory. Entrants can enter additional libraries one per line or import requirements.txt. Comments, on a line of their own or after a requirement (numpy==1.26 # for the metric), and blank lines do not count towards the 32-requirement limit; the whole request is limited to 8 KiB. The server checks compatibility, downloads PyPI wheels and verifies installation without network access. Base package versions cannot be replaced.

Choose a ready set before explicitly confirming notebook submission. Its packages and base image remain bound to that submission; changing the current selection does not alter a rerun. The set is installed into the submission's memory at every run and counts against its limit: it may take at most half, the rest being the kernel's, the data's and the working folder's. A bigger set cannot be chosen, and the entrant sees how much it would take. On the Docker path, preparation requires Docker Engine 28 or newer. Notebook execution and scoring have no internet access.

Checks and reruns

Every attempt has its own identifier and directories. Rescoring an active submission waits for that attempt to finish; an old CSV cannot become the result of a failed new run. Changes to the metric, data, base environment or starter notebook require an up-to-date baseline check. After changing notebook data or its environment, metric-only rescoring is insufficient: rerun the notebook.

Executor availability

Submissions and package preparation run where the rooms' kernels run: on the Docker path (the server image, colloq start) in throwaway containers on the same Docker, on k3s in Pods the broker starts. If the executor or package preparation is unavailable, the interface says why and refuses the work before it enters the queue; /api/health reports each capability separately.

+

Additional packages

The Packages tab shows the pinned Python version and complete base inventory. Entrants can enter additional libraries one per line or import requirements.txt. Comments, on a line of their own or after a requirement (numpy==1.26 # for the metric), and blank lines do not count towards the 32-requirement limit; the whole request is limited to 8 KiB. The server checks compatibility, downloads PyPI wheels and verifies installation without network access. Base package versions cannot be replaced.

Choose a ready set before explicitly confirming notebook submission. Its packages and base image remain bound to that submission; changing the current selection does not alter a rerun. The set is installed into the submission's memory at every run and counts against its limit: it may take at most half, the rest being the kernel's, the data's and the working folder's. A bigger set cannot be chosen, and the entrant sees how much it would take. On the Docker path, preparation requires Docker Engine 28 or newer. Notebook execution and scoring have no internet access.

Checks and reruns

Every attempt has its own identifier and directories. Rescoring an active submission waits for that attempt to finish; an old CSV cannot become the result of a failed new run. Changes to the metric, data, base environment or starter notebook require an up-to-date baseline check. After changing notebook data or its environment, metric-only rescoring is insufficient: rerun the notebook.

Executor availability

Submissions and package preparation run where the rooms' kernels run: on the Docker path (the server image, colloq start) in throwaway containers on the same Docker, on k3s in Pods the broker starts. If the executor or package preparation is unavailable, the interface says why and refuses the work before it enters the queue; /api/health reports each capability separately. A competition with GPU on also needs a share of a card: with no card given to submissions (on Kubernetes, no GPU resource set for them) it refuses uploads and the baseline check with that reason.

diff --git a/docs/pages/en/environments.html b/docs/pages/en/environments.html index 9845efca..a7121c95 100644 --- a/docs/pages/en/environments.html +++ b/docs/pages/en/environments.html @@ -5,7 +5,7 @@

Building a release (maintainers)

The Publi --k3s-version EXACT_TESTED_K3S_VERSION --environments base,kaggle-base

For GPU, also pass --gpu-toolkit-version EXACT_NVIDIA_APT_VERSION and --gpu-device-plugin-image NVIDIA_PLUGIN_IMAGE_AT_SHA256. Log in to the registry on the build machine. The k3s version must be v1.33 or newer: a room's memory is changed during class through the pods/resize subresource, and manifest validation rejects an older one. Publishing the artifacts is not, by itself, a test on the target VM.

Memory and CPU for a class

The Resources section is in the new class form and in the settings of an existing class (class menu → Settings). The Memory for the room's kernel, GB field moves in 0.5 GB steps, from 0.5 GB up to the machine's memory minus 1 GB. The Processor, cores field takes a whole number from 1 up to the machine's core count. To go back to the shared value, clear the field or click back to the environment default or back to the instance default.

The hint under the field shows the total memory, the free memory and the selected environment's default. If Docker runs in its own virtual machine, as Docker Desktop and Colima do on macOS, the hint starts with Rooms live in Docker. In that case the memory size, the core count and the fields' upper limits come from Docker, not from the computer. Free memory is Docker's memory minus 1 GB and minus the limits of rooms that are already running. If free memory cannot be determined, the hint leaves it out. The form does not stop you from setting a limit above free memory, but it warns that the kernel may not start. For a running room, only the increase over what it already has is compared with free memory. If the machine has an NVIDIA graphics card, the section shows it and whether the selected environment uses the card. GPU memory is not limited: all rooms on the card share it.

In Docker, a new memory limit applies to the running kernel immediately, without a restart; going back to the environment default takes effect the next time the room's container starts. A new core count also applies to the running kernel immediately, without a restart, and so does going back to the instance default. The numpy and torch thread count (OMP_NUM_THREADS and related variables) is set when the room's container is created and does not change for a running container, not even with a kernel restart: the restarted kernel inherits the same container's environment. The next container of the room gets the new thread count, for example when the room has stood empty for two hours and is opened again. Until then you can raise the thread count right in the notebook: threadpoolctl.threadpool_limits(n) for numpy and scikit-learn, torch.set_num_threads(n) for torch. Without KERNEL_MEM, the default is 4 GB, or 16 GB for a GPU environment. KERNEL_MEM sets the default for ordinary environments (a GPU environment stays at 16 GB), KERNEL_MEM_<ENVIRONMENT> sets it for one environment (for example, KERNEL_MEM_BASE_GPU=16g), and KERNEL_CPUS sets the number of cores (2 by default). The .env file that colloq start creates already contains KERNEL_MEM=4g. The owner can change these instance defaults without a restart (/api/admin/resources): a saved value wins over .env until it is cleared. The class's second container — the one where students' personal notebooks run — gets the same memory and CPU numbers, in a cgroup of its own, so a greedy draft does not take memory away from the lecture. It is never given a GPU. KERNEL_OWN_MAX (60 by default) sets how many kernels may live in it at once: over that, a run in a personal notebook is refused in words. Its process ceiling is its own, KERNEL_OWN_PIDS (2048 by default) — it holds dozens of kernels, and the room's 512 would run out at about thirty. A personal notebook's kernel that has computed nothing for KERNEL_OWN_IDLE_MIN minutes (30 by default; 0 never stops them) is stopped: the notebook is told in the kernel log, the next run brings it back, and the container goes with the last kernel in it.

A class's running container holds its whole memory limit even when it is nearly empty: while it runs, no other class can have that memory. A class with nobody inside and nothing computing lets its kernels go after KERNEL_ROOM_IDLE_MIN minutes (120 by default; 0 keeps them); the panel's Resources tab → «Default room» changes it too. The same tab's Running now section shows which classes hold memory, how much of it is really used and who is inside; the server owner can stop any class's kernels or all idle ones at once. A stopped kernel loses its variables, the files stay, and the next run brings it back. How this works in a k3s installation is described below.

Room resources

In a k3s installation, a room Pod gets 2Gi of memory, 2 CPUs and 2Gi of ephemeral storage by default; requests equal limits. The memory and the CPU count from the class's “Resources” section reach the Pod, and you can raise or lower them during class: the broker changes them on the running Pod in place, so the kernel does not restart and variables stay. The back to the environment default and back to the instance default buttons also apply at once. The Pod takes the numpy and torch thread count from its CPU limit when it starts and, as in Docker, does not change it for the running kernel, neither after an in-place change nor after a kernel restart; the room's next Pod gets the new count. The broker has one default for all environments, GPU ones included, and that is the default the form's hint shows. The field's upper limit is the node's memory minus 1 GB. The broker variables RUNTIME_KERNEL_MEMORY and RUNTIME_KERNEL_MEMORY_MAX set the default and the ceiling: put them in the same instance.env as the app settings (see installation configuration), for example RUNTIME_KERNEL_MEMORY=4Gi. Values are whole Mi or Gi from 64Mi to 256Gi, and the ceiling can't be below the default; the installer checks them before stopping rooms and keeps them across updates. Changing memory in place needs Kubernetes 1.33 or newer, and a release with an older k3s fails validation. These are resource limits, not a promise of a certain number of students or a certain model training speed.

A room's memory and cores are reserved in full, even if Python never touches them. That is why the hint counts as free the node's memory minus 1 GB and minus the memory already given to running rooms. If the node can't cover an increase in memory or cores right now, the new value is saved and waits: Kubernetes applies it on its own once the resource frees up, for example when a neighbouring room stops. The broker does not set more than the node can ever provide: the running Pod keeps its previous value. A new room has nowhere to wait: if the node can't cover its memory, CPU or GPU, startup stops after 20 seconds. The teacher sees in the room how much it asked for and what to lower or close; students see a short note that Python can't start right now and that the teacher can see why. If code runs out of memory, k3s ends the room's whole Pod, not just the kernel as in Docker; the next run starts a new Pod, already with the new memory value.

Memory in /dev/shm counts toward the Pod limit: 64 MiB by default for a CPU environment and 1 GiB for a GPU one. The installer sets a limit of 256 PIDs per Pod. A namespace ResourceQuota and any extra admission rules are up to the operator; no ResourceQuota ships with the installation. There is no enforced per-room file quota for the workspace; keep an eye on the separate file partition and on free space.

-

GPU environments

Each GPU room requests one nvidia.com/gpu and RuntimeClass nvidia. Device sharing is not turned on automatically. Several GPU rooms running at once need a matching number of available devices, or a sharing policy that has been separately designed and tested.

The installer keeps the host driver, installs pinned versions of the NVIDIA Container Toolkit and device plugin, and runs a real PyTorch CUDA operation as a check. The check needs a free GPU; a card that is in use is not taken away from a running class.

sudo scripts/cluster.sh gpu-preflight
+
GPU environments
Two GPU environments ship with Colloq: base-gpu, which is PyTorch and torchvision as CUDA wheels and nothing else, and kaggle-gpu on top of it, which is the kaggle-base set at the same pinned versions plus seaborn, OpenCV and timm. Neither image is published. On a machine of your own they are built where the card is: on first boot with KERNEL_PRELOAD=base,base-gpu,kaggle-gpu, or with Build in the Environments panel. base-gpu takes about 9.9 GB of disk, and kaggle-gpu about 2.8 GB more, 12.7 GB in all. Both carry torch 2.14.1 and torchvision 0.29.1, wheels built for CUDA 13.0, so the machine with the card needs an NVIDIA driver of 580 or newer: on an older one PyTorch imports but never sees the card (torch.cuda.is_available() is false). nvidia-smi shows the driver's version and the highest CUDA it supports in its header. For a release, name them in --environments together with the two GPU options above. A competition on either runs on the CPU until its GPU switch is on; see a card for submissions.
Each GPU room requests one nvidia.com/gpu and RuntimeClass nvidia. Device sharing is not turned on automatically. Several GPU rooms running at once need a matching number of available devices, or a sharing policy that has been separately designed and tested.
The installer keeps the host driver, installs pinned versions of the NVIDIA Container Toolkit and device plugin, and runs a real PyTorch CUDA operation as a check. The check needs a free GPU; a card that is in use is not taken away from a running class.
sudo scripts/cluster.sh gpu-preflight
 sudo k3s kubectl get nodes \
   -o "custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
An application that responds normally does not prove the GPU is ready. If the card is not allocated or CUDA does not work, find the cause before the class starts.

 
Keep old images
Historical catalog revisions are kept as long as rooms refer to them. Do not delete the corresponding images from the registry. An application backup contains the catalog and the digests, but not the image layers.

diff --git a/docs/pages/en/install.html b/docs/pages/en/install.html
index c0bd6e3c..27c014b0 100644
--- a/docs/pages/en/install.html
+++ b/docs/pages/en/install.html
@@ -1,5 +1,5 @@
 
-
What you need
A k3s installation is designed for one Linux amd64 VM with systemd and IPv4. You need root access, curl, Python 3, iptables (plus ip6tables on a host with IPv6) and the standard Linux utilities. A GPU release needs Debian or Ubuntu with apt, the gnupg package and a working NVIDIA driver installed in advance: the installer does not replace it. You do not need to install Node.js or the Docker daemon on the server: the application and the kernels run on k3s/containerd.
Set aside resources for k3s, the application and every room that runs at the same time. The number of connected students does not determine the load by itself: what matters is the code, memory, files and the number of active kernels. See resource planning.

+
What you need
A k3s installation is designed for one Linux amd64 VM with systemd and IPv4. You need root access, curl, Python 3, iptables (plus ip6tables on a host with IPv6) and the standard Linux utilities. A GPU release needs Debian or Ubuntu with apt, the gnupg package and a working NVIDIA driver of 580 or newer installed in advance: the installer does not replace it, and on an older driver the GPU environments' torch 2.14.1, built for CUDA 13.0, never sees the card. You do not need to install Node.js or the Docker daemon on the server: the application and the kernels run on k3s/containerd.
Set aside resources for k3s, the application and every room that runs at the same time. The number of connected students does not determine the load by itself: what matters is the code, memory, files and the number of active kernels. See resource planning.

 
Verify the release kit
Get the artifacts through a trusted channel. Verify the checksums, unpack the tools archive and work from its directory. The manifest records the source commit, the real digests of the application, broker and environment images, the exact k3s version and the hashes of the tools.
sha256sum -c SHA256SUMS
 tar -xzf colloq-deploy.tar.gz
 python3 scripts/release.py validate --release release.json
diff --git a/docs/pages/en/kubernetes.html b/docs/pages/en/kubernetes.html
index e1cbe580..4c60655d 100644
--- a/docs/pages/en/kubernetes.html
+++ b/docs/pages/en/kubernetes.html
@@ -1,8 +1,8 @@
 
In short for the platform team
What it isColloq is a collaborative Jupyter classroom: a web app, a runtime broker and a separate Pod with a Python kernel for each room. It goes into your cluster as a Helm chart, oci://ghcr.io/colloq-edu/charts/colloq, in one namespace. Open source, MIT license.
What runsThe app Deployment, one replica: the database is SQLite. The broker Deployment, one replica: the only thing that talks to the Kubernetes API. A Pod and a Service for each running room. A second Pod and Service per room for students' personal notebooks. Short-lived competition Pods: submissions, the metric, package preparation. Room and competition Pods are created by the broker, not by the chart.
What it needsOne namespace with namespace-admin rights; the broker's Role from Permissions; a StorageClass — ReadWriteOnce required, ReadWriteMany preferred; an ingress controller and a certificate; a registry mirror with the release's images.
What it never needscluster-admin or cluster-scoped objects: Namespace, PersistentVolume, ClusterRole, CRD, DaemonSet, RuntimeClass. Privileged containers, hostPath, hostNetwork or hostPID. Root: every container runs as uid 1000. Internet access from the cluster.
NetworkInbound: only the ingress controller to the app, port 3000. Inside the namespace: app → broker, app and broker → rooms. Outbound: broker → Kubernetes API, Pods → cluster DNS; optionally app → the AI model and an outbound proxy. Students' code gets DNS only; competition submissions get nothing.
DataTwo PVCs: colloq-data — the SQLite database, keys, class page files, competition data and submissions, database snapshots; colloq-workspace — room files.
BackupsVolume snapshots with your own tools (Velero, CSI), plus consistent database snapshots the app writes into the colloq-data volume itself once a day.

 
Students' and teachers' browsersThe cluster's ingress controller, TLSApp · Deployment colloq-app, 1 replicaBroker · Deployment colloq-runtime, 1 replica
Room A · Pod and ServiceRoom B · Pod and ServiceThe room's personal notebooksCompetition submission
The app — a Node.js server with the built web interface and an SQLite database — keeps everything on two volumes and never calls the Kubernetes API. When someone joins a room or runs a cell, the app asks the broker for the room's Pod. The broker creates the Pod and a ClusterIP Service from a template fixed in its code, waits for Jupyter and hands the app its address and token; from then on the app talks to the room's Jupyter directly, on port 8888. A room's Pod lives while anyone is in the room and for 2 hours after everyone has left. Room Pods survive restarts and upgrades of the app and the broker: the new broker finds them by their labels, and Python variables stay.

-
What we assume about your cluster
This is a checklist for your team: go through it before installing. The answer to each item is either "yes, that's us" or a chart value to set.
WhatWe assumeIf yours differs
Kubernetes version1.30 or newer, upstream-compatible: vanilla, Deckhouse and the like. Changing a running room's memory and cores without a restart needs 1.33 or newer and the pods/resize permission.Older than 1.33, or no such permission: runtime.inPlaceResize: false. The rule is dropped from the Role, resizing a live room answers "unsupported" rather than failing, and the new number goes to the room's next Pod.
NamespaceWe get one namespace with namespace-admin rights, and it holds one Colloq installation: object names are fixed (colloq-app, colloq-runtime, …). The chart creates no cluster-scoped objects.Your team creates the namespace, its labels and its quota. The chart creates the Role and RoleBinding itself, and Kubernetes won't let anyone grant a verb they don't hold: whoever installs the chart needs every verb in Permissions.
Pod Security and policiesPod Security Admission restricted is enforced, and Kyverno or Gatekeeper require requests and limits on every container, probes, runAsNonRoot and readOnlyRootFilesystem, forbid privileged, hostPath and hostNetwork, admit images only from the internal registry and only by digest, and require the standard app.kubernetes.io/* labels.Every Colloq Pod passes restricted; details and exceptions are under Security. commonLabels adds your labels to every chart object and every broker Pod; rooms.podLabels and rooms.podAnnotations add them to broker Pods only.
Network policiesNetworkPolicy is enforced (Calico, Cilium), and the namespace may be default-deny.The chart declares every flow it needs, ingress and egress; they are listed under Network. Say where the ingress controller lives (networkPolicy.ingressController.from), where cluster DNS lives (networkPolicy.dns.to), and the API servers' addresses (networkPolicy.kubeApiServer).
Ingress and TLSIngress through ingress-nginx, with a configurable class and annotations. TLS: the organisation's certificate from an existing Secret or from cert-manager. Users come from the corporate network, over VPN or, when students are external, from the internet.Another controller: ingress.className and ingress.annotations with the same meaning — request bodies of 256 MB or more, timeouts of an hour or more, no response buffering. Or ingress.enabled: false: publish the Service colloq-app:3000 with your own object and set config.publicUrl.
Sign-inStudents join with the class link, teachers sign in with a personal link. A proxy with SSO that forwards a signed JWT — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — may stand in front of Colloq.config.sso.*: teachers sign in with their corporate account, students who have one join under their own name; see Sign-in through your SSO. Without such a proxy, nothing to set.
StorageA StorageClass with ReadWriteOnce always exists; one with ReadWriteMany (CephFS) may.By default the chart assumes ReadWriteOnce: everything that mounts Colloq's volumes runs on one node. With ReadWriteMany, set persistence.accessMode: ReadWriteMany and room Pods spread over the cluster; see Storage.
No internetThe cluster has no internet access. Images come through a registry mirror (Harbor, Nexus), PyPI packages through a Nexus mirror, and the Oracle through an internal model with an OpenAI-compatible API, or it stays off. There may be an outbound HTTP proxy and an internal CA.global.imageRegistry — the mirror for every image, kernel images included; dependencies.indexUrl and dependencies.mirrorEgress — the PyPI mirror; config.ai.baseUrl — the model; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. See Images and the registry mirror.
Secrets and GitOpsSecrets come from Vault through External Secrets, and deployment goes through ArgoCD.Every secret can come from an existing Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. With those set, the chart renders no random value, so ArgoCD sees no perpetual drift.
Monitoring and logsPrometheus Operator; logs are collected from stdout and stderr.metrics.enabled turns on /metrics with a token, metrics.serviceMonitor.enabled adds a ServiceMonitor; without the Prometheus Operator, scrape /metrics your own way. config.logFormat: json writes one JSON object per line.
BackupsVolume snapshots: Velero, CSI.The app itself writes consistent database snapshots into the colloq-data volume (config.dbSnapshotHours, config.dbSnapshotKeep), so a volume snapshot always carries a whole copy of the database.
GPUGPU nodes are optional.GPU rooms take a RuntimeClass, a nodeSelector and tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations; see Operations.

+
What we assume about your cluster
This is a checklist for your team: go through it before installing. The answer to each item is either "yes, that's us" or a chart value to set.
WhatWe assumeIf yours differs
Kubernetes version1.30 or newer, upstream-compatible: vanilla, Deckhouse and the like. Changing a running room's memory and cores without a restart needs 1.33 or newer and the pods/resize permission.Older than 1.33, or no such permission: runtime.inPlaceResize: false. The rule is dropped from the Role, resizing a live room answers "unsupported" rather than failing, and the new number goes to the room's next Pod.
NamespaceWe get one namespace with namespace-admin rights, and it holds one Colloq installation: object names are fixed (colloq-app, colloq-runtime, …). The chart creates no cluster-scoped objects.Your team creates the namespace, its labels and its quota. The chart creates the Role and RoleBinding itself, and Kubernetes won't let anyone grant a verb they don't hold: whoever installs the chart needs every verb in Permissions.
Pod Security and policiesPod Security Admission restricted is enforced, and Kyverno or Gatekeeper require requests and limits on every container, probes, runAsNonRoot and readOnlyRootFilesystem, forbid privileged, hostPath and hostNetwork, admit images only from the internal registry and only by digest, and require the standard app.kubernetes.io/* labels.Every Colloq Pod passes restricted; details and exceptions are under Security. commonLabels adds your labels to every chart object and every broker Pod; rooms.podLabels and rooms.podAnnotations add them to broker Pods only.
Network policiesNetworkPolicy is enforced (Calico, Cilium), and the namespace may be default-deny.The chart declares every flow it needs, ingress and egress; they are listed under Network. Say where the ingress controller lives (networkPolicy.ingressController.from), where cluster DNS lives (networkPolicy.dns.to), and the API servers' addresses (networkPolicy.kubeApiServer).
Ingress and TLSIngress through ingress-nginx, with a configurable class and annotations. TLS: the organisation's certificate from an existing Secret or from cert-manager. Users come from the corporate network, over VPN or, when students are external, from the internet.Another controller: ingress.className and ingress.annotations with the same meaning — request bodies of 256 MB or more, timeouts of an hour or more, no response buffering. Or ingress.enabled: false: publish the Service colloq-app:3000 with your own object and set config.publicUrl.
Sign-inStudents join with the class link, teachers sign in with a personal link. A proxy with SSO that forwards a signed JWT — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — may stand in front of Colloq.config.sso.*: teachers sign in with their corporate account, students who have one join under their own name; see Sign-in through your SSO. Without such a proxy, nothing to set.
StorageA StorageClass with ReadWriteOnce always exists; one with ReadWriteMany (CephFS) may.By default the chart assumes ReadWriteOnce: everything that mounts Colloq's volumes runs on one node. With ReadWriteMany, set persistence.accessMode: ReadWriteMany and room Pods spread over the cluster; see Storage.
No internetThe cluster has no internet access. Images come through a registry mirror (Harbor, Nexus), PyPI packages through a Nexus mirror, and the Oracle through an internal model with an OpenAI-compatible API, or it stays off. There may be an outbound HTTP proxy and an internal CA.global.imageRegistry — the mirror for every image, kernel images included; dependencies.indexUrl and dependencies.mirrorEgress — the PyPI mirror; config.ai.baseUrl — the model; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. See Images and the registry mirror.
Secrets and GitOpsSecrets come from Vault through External Secrets, and deployment goes through ArgoCD.Every secret can come from an existing Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. With those set, the chart renders no random value, so ArgoCD sees no perpetual drift.
Monitoring and logsPrometheus Operator; logs are collected from stdout and stderr.metrics.enabled turns on /metrics with a token, metrics.serviceMonitor.enabled adds a ServiceMonitor; without the Prometheus Operator, scrape /metrics your own way. config.logFormat: json writes one JSON object per line.
BackupsVolume snapshots: Velero, CSI.The app itself writes consistent database snapshots into the colloq-data volume (config.dbSnapshotHours, config.dbSnapshotKeep), so a volume snapshot always carries a whole copy of the database.
GPUGPU nodes are optional.GPU rooms take a RuntimeClass, a nodeSelector and tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations. Competition submissions get a GPU only through gpu.competitions.resource, the device-plugin resource they request; see Operations.

 
Permissions
Only the broker has rights in the cluster: the Role and RoleBinding colloq-runtime in this namespace. No ClusterRole, no access to nodes, Secrets, Deployments or other namespaces.
ResourceVerbsWhy
podsget, list, create, deleteRoom, personal-notebook and competition Pods. list is for the census: a restarted broker finds the running Pods by their labels and adopts them instead of creating them again.
servicesget, list, create, deleteA ClusterIP Service for each room and its personal notebooks — the stable name the app reaches Jupyter by — and for the package-preparation proxy. A room deleted for good leaves a tombstone: a headless Service with no selector and no ClusterIP, so that its ID can't be opened again.
pods/resizepatchChange a running room's memory and cores without a restart (Kubernetes 1.33+): the patch touches only memory and cpu of the kernel container. Optional: with runtime.inPlaceResize: false the rule is gone.
persistentvolumeclaimsgetBefore a competition, check that the data volume is in place.
networkpolicies (networking.k8s.io)getBefore a competition, check that the submissions' isolation policies exist. Without them submissions are refused, and the teacher sees why.
There is no patch or update on Pods and Services: the broker doesn't edit live objects — it changes only a room's resources through pods/resize, and deletes and recreates everything else, with a UID precondition. RBAC can't constrain the fields of a Pod, so the Pod templates are fixed in the broker's code, and its HTTP API takes only a room ID, an environment from the catalog, an image revision, memory and cores: no arbitrary template, no hostPath, no image outside the catalog. Restricted Pod Security and your policies hold the rest.
ServiceAccountAPI tokenWho
colloq-appnoThe app. It never calls the Kubernetes API.
colloq-runtimeyes: the broker's Pod mounts a projected token that the kubelet rotatesThe broker; bound to the Role above.
colloq-kernelnoRoom and personal-notebook Pods: students' code holds no API credentials.
the namespace defaultnoCompetition Pods: no token is mounted.
All three ServiceAccounts set automountServiceAccountToken: false; only the broker's Pod turns the token on.

-
Security
Pod Security. Every Colloq Pod — the app, the broker, rooms, personal notebooks, competitions — passes the restricted profile: runAsNonRoot, uid and gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Only volumes are writable: emptyDir for /tmp, the home directory and /dev/shm (in memory: 64 MiB, 1 GiB for GPU rooms), and the PVCs. Your team puts the pod-security.kubernetes.io/enforce: restricted label on the namespace: the chart doesn't own it.
Policy engines.
  • Requests and limits are set on every container, including the init container and the sidecar of competition Pods. A room kernel's requests equal its limits (QoS Guaranteed): memory and cores are reserved in full, plus 2 GiB of ephemeral storage.
  • Probes. The app: HTTP startupProbe and livenessProbe on /api/livez, readinessProbe on /api/readyz. The broker: TCP 8787. Room and personal-notebook kernels: TCP readiness and liveness probes on 8888; the liveness probe acts only after five minutes of refused connections, since a Pod with restartPolicy: Never ends with the class's variables when it does. Submission containers run to completion and have no probes; their exporter and the package proxy have readiness probes. If a probe rule doesn't fit run-to-completion Pods, exempt them by the label app.kubernetes.io/managed-by: colloq-runtime.
  • Labels. Chart objects carry app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm and helm.sh/chart. Pods the broker creates carry app.kubernetes.io/name, instance and part-of, app.kubernetes.io/managed-by: colloq-runtime and colloq.dev/role: kernel (the personal-notebook Pod also has colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. The app.kubernetes.io/* and colloq.* keys belong to the chart and the broker and can't be overridden.
  • Digests. The chart always renders the app and broker images as registry/path@sha256:… — never a tag, never latest. Kernel images in the catalog are accepted only in the same form: neither the chart nor the broker lets a tag-only reference through.
  • Registry. global.imageRegistry rewrites the registry of every image, including the catalog's kernels, the competition exporter and the package proxy.
  • Mutating webhooks. After creating a room's Pod the broker checks it against what it asked for. If a webhook changed its containers, volumes or security settings — injected a sidecar, rewrote imagePullPolicy — the broker refuses the Pod and the room doesn't start; the broker's log names the fields that changed. Exempt Pods labelled app.kubernetes.io/managed-by: colloq-runtime from such webhooks; for Istio, rooms.podLabels: {sidecar.istio.io/inject: "false"} is enough. Placement the cluster fills in by itself — PodNodeSelector, PodTolerationRestriction, priority, RuntimeClass, Kueue — doesn't disturb the check.
What code runs as whom.
PodImage and codeAPI tokenVolumesNetwork
Appcolloq-app: the Colloq server on Node.jsnocolloq-data and colloq-workspacein from the ingress controller; out to the broker, rooms, DNS and, optionally, the AI model and a proxy
Brokercolloq-runtime: the broker on Node.jsyesnonein from the app; out to the Kubernetes API, rooms, submissions and DNS
Roomthe room environment's colloq-kernel: Jupyter and students' codenoits own folder of colloq-workspacein from the app and the broker on 8888; out: DNS only
Personal notebookscolloq-kernel: students' code in personal notebooks, no GPUnoits room's folder, at the same pathas a room
Submissioncolloq-kernel: the entrant's notebook or the metric; colloq-runtime: the result exporternofolders of colloq-data, inputs read-onlynone; in: only the broker to the exporter on 8765
Package preparationcolloq-kernel: pip downloads entrants' own packagesnofolders of colloq-dataonly to its own proxy on 3128, and DNS
Package proxycolloq-runtime: a CONNECT proxynononein from preparation; out: DNS and the package index — public PyPI on 443 or your mirror
Isolation between rooms. Each room has a Pod of its own: its own processes, its own emptyDir, its own Jupyter token — the broker derives it from the room secret with an HMAC and gives it to the app only. The Pod mounts only its own folder of the colloq-workspace volume (subPath); other rooms' files aren't in it. The network policy lets only the app and the broker reach a room's port 8888 and lets nothing out of a room but DNS: neighbouring rooms, the app and the API are out of reach of students' code. Inside a room, participants share the kernel, the files and the Linux user — by design: it is a shared workspace. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node: test it from a room (see the checks after installing) and close it with your CNI's host policies if needed.
Secrets.
SecretWhat it holdsWho reads it
config.existingSecret or colloq-app-configKeys are app settings: OPENAI_API_KEY, SESSION_SECRET, a proxy with a password in HTTPS_PROXY, or any other settingthe app
colloq-runtime-auth, key runtime-tokenThe Bearer token the app uses with the brokerthe app and the broker
colloq-room-secret, key room-secretThe key the broker derives rooms' Jupyter tokens and submission export tokens fromthe broker only
colloq-metrics, key metrics-tokenThe Bearer token for /metrics, when metrics are onthe app and Prometheus
global.imagePullSecretsThe mirror's .dockerconfigjson; broker Pods get the first one toothe kubelet
Each can come from an existing Secret — for example one External Secrets creates from Vault. When the broker token, the room secret or the metrics token are not given, the chart generates them: under the helm CLI once, reading them back from the cluster with lookup afterwards. ArgoCD and other tools that render the chart with helm template can't read them back and would generate new values on every sync, restarting rooms — set existingSecret there. The broker token and the room secret are 43 to 128 characters of [A-Za-z0-9_-] from at least 32 random bytes: openssl rand -hex 32. Keep the secrets stable: a new SESSION_SECRET invalidates every link and sign-in handed out, and a new room secret replaces room Pods at their next start, losing Python variables. Without SESSION_SECRET, the app generates one once and keeps it in the colloq-data volume. When you move from a generated secret to existingSecret, copy its value into the new Secret first: the chart deletes its own.
Secrets live in the colloq-data volume too: the setup token, the teachers' sign-in keys in the database, the model key if it was entered in the panel. Keep snapshots of that volume as private as Secrets.

+
Security
Pod Security. Every Colloq Pod — the app, the broker, rooms, personal notebooks, competitions — passes the restricted profile: runAsNonRoot, uid and gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Only volumes are writable: emptyDir for /tmp, the home directory and /dev/shm (in memory: 64 MiB, 1 GiB for GPU rooms, half the memory of a GPU submission and at most gpu.competitions.shmMb), and the PVCs. Your team puts the pod-security.kubernetes.io/enforce: restricted label on the namespace: the chart doesn't own it.
Policy engines.
  • Requests and limits are set on every container, including the init container and the sidecar of competition Pods. A room kernel's requests equal its limits (QoS Guaranteed): memory and cores are reserved in full, plus 2 GiB of ephemeral storage.
  • Probes. The app: HTTP startupProbe and livenessProbe on /api/livez, readinessProbe on /api/readyz. The broker: TCP 8787. Room and personal-notebook kernels: TCP readiness and liveness probes on 8888; the liveness probe acts only after five minutes of refused connections, since a Pod with restartPolicy: Never ends with the class's variables when it does. Submission containers run to completion and have no probes; their exporter and the package proxy have readiness probes. If a probe rule doesn't fit run-to-completion Pods, exempt them by the label app.kubernetes.io/managed-by: colloq-runtime.
  • Labels. Chart objects carry app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm and helm.sh/chart. Pods the broker creates carry app.kubernetes.io/name, instance and part-of, app.kubernetes.io/managed-by: colloq-runtime and colloq.dev/role: kernel (the personal-notebook Pod also has colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. The app.kubernetes.io/* and colloq.* keys belong to the chart and the broker and can't be overridden.
  • Digests. The chart always renders the app and broker images as registry/path@sha256:… — never a tag, never latest. Kernel images in the catalog are accepted only in the same form: neither the chart nor the broker lets a tag-only reference through.
  • Registry. global.imageRegistry rewrites the registry of every image, including the catalog's kernels, the competition exporter and the package proxy.
  • Mutating webhooks. After creating a room's Pod the broker checks it against what it asked for. If a webhook changed its containers, volumes or security settings — injected a sidecar, rewrote imagePullPolicy — the broker refuses the Pod and the room doesn't start; the broker's log names the fields that changed. Exempt Pods labelled app.kubernetes.io/managed-by: colloq-runtime from such webhooks; for Istio, rooms.podLabels: {sidecar.istio.io/inject: "false"} is enough. Placement the cluster fills in by itself — PodNodeSelector, PodTolerationRestriction, priority, RuntimeClass, Kueue — doesn't disturb the check.
What code runs as whom.
PodImage and codeAPI tokenVolumesNetwork
Appcolloq-app: the Colloq server on Node.jsnocolloq-data and colloq-workspacein from the ingress controller; out to the broker, rooms, DNS and, optionally, the AI model and a proxy
Brokercolloq-runtime: the broker on Node.jsyesnonein from the app; out to the Kubernetes API, rooms, submissions and DNS
Roomthe room environment's colloq-kernel: Jupyter and students' codenoits own folder of colloq-workspacein from the app and the broker on 8888; out: DNS only
Personal notebookscolloq-kernel: students' code in personal notebooks, no GPUnoits room's folder, at the same pathas a room
Submissioncolloq-kernel: the entrant's notebook or the metric; colloq-runtime: the result exporternofolders of colloq-data, inputs read-onlynone; in: only the broker to the exporter on 8765
Package preparationcolloq-kernel: pip downloads entrants' own packagesnofolders of colloq-dataonly to its own proxy on 3128, and DNS
Package proxycolloq-runtime: a CONNECT proxynononein from preparation; out: DNS and the package index — public PyPI on 443 or your mirror
Isolation between rooms. Each room has a Pod of its own: its own processes, its own emptyDir, its own Jupyter token — the broker derives it from the room secret with an HMAC and gives it to the app only. The Pod mounts only its own folder of the colloq-workspace volume (subPath); other rooms' files aren't in it. The network policy lets only the app and the broker reach a room's port 8888 and lets nothing out of a room but DNS: neighbouring rooms, the app and the API are out of reach of students' code. Inside a room, participants share the kernel, the files and the Linux user — by design: it is a shared workspace. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node: test it from a room (see the checks after installing) and close it with your CNI's host policies if needed.
Secrets.
SecretWhat it holdsWho reads it
config.existingSecret or colloq-app-configKeys are app settings: OPENAI_API_KEY, SESSION_SECRET, a proxy with a password in HTTPS_PROXY, or any other settingthe app
colloq-runtime-auth, key runtime-tokenThe Bearer token the app uses with the brokerthe app and the broker
colloq-room-secret, key room-secretThe key the broker derives rooms' Jupyter tokens and submission export tokens fromthe broker only
colloq-metrics, key metrics-tokenThe Bearer token for /metrics, when metrics are onthe app and Prometheus
global.imagePullSecretsThe mirror's .dockerconfigjson; broker Pods get the first one toothe kubelet
Each can come from an existing Secret — for example one External Secrets creates from Vault. When the broker token, the room secret or the metrics token are not given, the chart generates them: under the helm CLI once, reading them back from the cluster with lookup afterwards. ArgoCD and other tools that render the chart with helm template can't read them back and would generate new values on every sync, restarting rooms — set existingSecret there. The broker token and the room secret are 43 to 128 characters of [A-Za-z0-9_-] from at least 32 random bytes: openssl rand -hex 32. Keep the secrets stable: a new SESSION_SECRET invalidates every link and sign-in handed out, and a new room secret replaces room Pods at their next start, losing Python variables. Without SESSION_SECRET, the app generates one once and keeps it in the colloq-data volume. When you move from a generated secret to existingSecret, copy its value into the new Secret first: the chart deletes its own.
Secrets live in the colloq-data volume too: the setup token, the teachers' sign-in keys in the database, the model key if it was entered in the panel. Keep snapshots of that volume as private as Secrets.

 
Network
Every flow Colloq needs. The chart declares them as network policies, ingress and egress, so the installation works in a default-deny namespace too.
FromToPortWhy
Ingress controllerAppTCP 3000All user traffic: pages, API, WebSocket, server-sent events. networkPolicy.ingressController.from says who is admitted (by default the ingress-nginx Pods in the ingress-nginx namespace). A controller on the host network arrives from node addresses: give an ipBlock of your nodes then.
AppBrokerTCP 8787Start, resize and stop a room's Pod, start a submission. Requests carry a Bearer token.
AppRoom and personal-notebook PodsTCP 8888Jupyter: running cells, the terminal. With the room's token.
BrokerRoom and personal-notebook PodsTCP 8888Check that Jupyter is up and accepts the token.
BrokerSubmission and package-preparation PodsTCP 8765Collect the result from the exporter.
BrokerKubernetes API server443 and 6443Create and delete Pods and Services. The network policy sees the address after DNAT, that is the API servers' own addresses, so by default any address on these ports is allowed. Narrow networkPolicy.kubeApiServer.cidrs to the output of kubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes. On Cilium the API server has an identity of its own that an ipBlock doesn't match: turn on networkPolicy.kubeApiServer.ciliumEntity, and the chart adds a CiliumNetworkPolicy in this namespace.
Every Colloq Pod except submissionsCluster DNSUDP and TCP 53Service names. networkPolicy.dns.to says where DNS is (by default k8s-app: kube-dns in kube-system). Rooms can lose DNS too: rooms.network: none.
Package preparationIts own package proxyTCP 3128pip goes only through a proxy that admits one package index.
Package proxyPyPI mirror or public PyPI443 or the mirror's portIndex and files of competition entrants' own packages. A policy knows addresses, not names: the mirror's addresses and ports go into dependencies.mirrorEgress. In a closed network turn off the general rule for public addresses on 443: dependencies.publicEgress: false.
AppAI model, outbound proxy, GitHubthe model's or proxy's portOptional: the Oracle and notebook import from GitHub. By default the app may go nowhere outside: addresses and ports go into networkPolicy.app.egress.
PrometheusAppTCP 3000, /metricsOptional, with metrics.enabled: networkPolicy.monitoring.from says who is admitted.
Nodes (kubelet)Registry mirrorTCP 443Images. Not described by network policies.
If the cluster runs NodeLocal DNSCache, DNS queries go to a node-local address rather than to the kube-dns Pods: add that address to networkPolicy.dns.to, for example ipBlock: {cidr: 169.254.20.10/32}.
Students' code doesn't get out: room and personal-notebook Pods have no egress but DNS, and rooms.network: none takes that away too. So pip install in a cell doesn't work: libraries come in kernel images; see Images and the registry mirror. Competition submissions get no network at all, not even DNS, and that is checked before any code runs: an init container tries to get out and lets the submission run only after three refusals in a row; if the network answers, the submission fails.
The client address: TRUSTED_PROXIES=private. Per-address caps — 60 newcomers to a room per 10 minutes, 20 Oracle questions a minute, competition sign-ins and joins — must see the student's address, not the ingress controller's. The app believes X-Forwarded-For only from connections whose address is in TRUSTED_PROXIES; the chart sets private (config.inbound.trustedProxies) — every private range, wherever the controller's Pod comes from. That is safe because only the ingress controller is admitted to the app's port 3000: the network policy lets no one else in, students' code included, so nobody inside the cluster can forge the header. The client is the rightmost address in the header that is not itself trusted; when every hop is trusted — and on a corporate network users' addresses are private too — the leftmost stands, which is the one the controller wrote. So the controller must replace the header rather than append its own address to one the client sent: ingress-nginx does that by default, while use-forwarded-headers and compute-full-forwarded-for are off. If a load balancer in front of the controller rewrites the source address, every student looks like one address: preserve the client address (externalTrafficPolicy: Local on the controller's Service, or PROXY protocol). If many people come from one address — through a NAT or a VPN concentrator — list it in config.inbound.sharedAddresses: per-address caps don't apply to it, while per-room and per-person caps still do.
ingress-nginx. The chart puts these annotations on its Ingress (ingress.annotations; a key set to null drops an annotation):
nginx.ingress.kubernetes.io/proxy-body-size: 256m       # competition data: up to 200 MB in one request
 nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"  # WebSocket and server-sent events live for hours
 nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
@@ -32,7 +32,7 @@ 
Sign-in through your SSO
If a proxy in front of Colloq check
       - cidrs: [10.20.0.15/32]              # the Teleport proxy, for the keys
         ports: [443]
Instead of fetching the keys over the network, put the JWKS document in config.sso.jwks: the chart mounts it from the ConfigMap colloq-sso-jwks and no egress is needed, but a key rotation in Teleport needs a new value. The header is Teleport-Jwt-Assertion by default; config.sso.header sets another. Authorization won't do: the browser sends the room token in it, and a proxy writing its own there would break files, history and the Oracle. oauth2-proxy with --pass-access-token passes the token in X-Forwarded-Access-Token — fine when the IdP issues access tokens as JWTs (Keycloak does); the audience is then whatever the IdP writes into the access token.
Token fields. By default: who it is — sub or username; the name — name, traits.name, traits.full_name, traits.display_name, username; the email — email, traits.email, traits.mail, username; roles — roles, groups, traits.groups. config.sso.claims.* sets other paths, and the first non-empty one wins. If the token carries only a login, config.sso.emailDomain turns it into the address teachers are recognised by. A line at start says where the keys come from and whom Colloq adds; a refused token is logged once a minute with the reason.
Students without an account. Only someone with an account gets through Teleport. If students have none, keep the ordinary way in for them: an Ingress to the same Service at a public address and the class links, with config.publicUrl set to that public address, which links are built from. Teachers then open both the panel and the rooms at the Teleport address — the panel still builds the links for students from the public one: the token is checked the same way wherever the request came from, and a page at the Teleport address passes the origin check when the Teleport agent is in config.inbound.trustedProxies (the chart sets private).
What stays with the proxy. A token is a pass until it expires: if Colloq can also be reached around the proxy, a stolen token works until its exp. Keep the token lifetime short, and keep the token header out of access logs.

 
Storage
PVCWhat it holdsWho mounts itDefault size
colloq-dataThe SQLite database (colloq.db with its WAL journal), the link-signing key, the setup token, output images, class page files in page-files/, competition data, hidden answers and submissions, package sets, database snapshots in snapshots/the app; competition Pods — their own folders10 GiB; more for competitions with large data, and about 0.3–0.8 GiB for each 33-class course with class pages
colloq-workspaceRoom files, a <room>/ folder eachthe app; a room's Pod and its personal-notebook Pod — only that room's folder100 GiB
The storage class, size and access mode are set for both volumes at once (persistence.storageClass, persistence.accessMode) or for each (persistence.data.*, persistence.workspace.*); existingClaim attaches a PVC you already have. With persistence.retain: true, the default, the PVCs survive both helm uninstall and deleting the ArgoCD application.
ReadWriteMany: preferred. With RWX on both volumes, room Pods spread over the cluster wherever the scheduler puts them, and the installation's capacity is bounded by the quota rather than by one node. CephFS or NFSv4, where the protocol itself handles file locking, will do; SQLite has a single writer, the app. SQLite's own documentation warns that locking is unreliable on many NFS implementations: if in doubt, keep the database volume on block storage (persistence.data.accessMode: ReadWriteOnce), which brings back running every Pod with a volume on one node.
ReadWriteOnce: one node. This is the chart's default. An RWO volume attaches to one node, so everything that mounts Colloq's volumes must run there: if at least one of the volumes is RWO, the chart sets RUNTIME_COLOCATE_WITH_APP=1 for the broker, and every Pod that mounts a volume — rooms, personal notebooks, submissions, package preparation — gets a required podAffinity to the app's Pod (label colloq.dev/role: app, topology kubernetes.io/hostname). The app's own Pod prefers the node where rooms already run: after an upgrade the new Pod needs exactly the node the volume is attached to. The broker mounts no volume and can run anywhere. What follows: the whole installation's capacity is one node's free resources; a room beyond them doesn't start, and after 20 seconds of waiting the teacher sees what was short — memory, cores or a GPU; maintenance on that node is a break for all of Colloq. Pick a node with headroom; see Resources and quotas.
File ownership. Every Pod writes as uid and gid 1000. The app's Pod sets fsGroup: 1000, and a fresh block volume becomes writable for group 1000 by itself. For RWX volumes the CSI driver doesn't apply fsGroup by default (fsGroupPolicy: ReadWriteOnceWithFSType covers only RWO volumes with a filesystem type): the root of such a volume must be writable by uid 1000 on the storage side.
No per-room disk quota. The colloq-workspace volume is shared: a student who fills it stops file saving for everyone. Watch PVC usage — the kubelet metrics kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes, or colloq_disk_free_bytes from /metrics — with an alert at 80 %. If the StorageClass allows volume expansion, the PVCs can grow.

-
Images and the registry mirror
ImageWhat it is
ghcr.io/colloq-edu/colloq-app:vX.Y.ZThe app.
ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZThe broker; also the submissions' result exporter and the package proxy.
ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseRoom and submission kernels: the base and kaggle-base environments, CPU only.
Every image is built for amd64 only. A release and its chart share one version number. In the published chart the defaults carry the digest of every image of the release: the app's and the broker's in image.app.digest and image.runtime.digest, the kernels' in the environment catalog catalog.environments. The same values come as a file of their own, values-X.Y.Z.yaml, attached to the release on GitHub: handy for copying the images into a mirror and for diffing releases in a GitOps repository. The chart in the source tree carries no digests and refuses to render without them.
The environment chosen for a class is a catalog entry: a name, an image by digest, a GPU flag. An environment of your own — other libraries, or a GPU — is built outside the cluster from the repository's kernel/ directory with KERNEL_ENV=<name>, pushed to your registry and added to catalog.environments: on Kubernetes the panel doesn't build environments. The base-gpu image (about 9 GB of CUDA wheels) is not published; build a GPU environment the same way.
The registry mirror. global.imageRegistry replaces the registry in every image reference — the app's, the broker's, the catalog's kernels, the competition exporter and package proxy — and keeps the path and the digest. The value may carry a path too: a Harbor proxy cache of ghcr.io in a project called ghcr works as is, and harbor.corp.example/ghcr gives harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. So the mirror must hold the images under the same paths. Pull secrets go into global.imagePullSecrets: the app's and the broker's Pods get them, and every Pod the broker creates gets the first one.
Without internet access. On a machine that can reach ghcr.io:
# the release's chart
+
Images and the registry mirror
ImageWhat it is
ghcr.io/colloq-edu/colloq-app:vX.Y.ZThe app.
ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZThe broker; also the submissions' result exporter and the package proxy.
ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseRoom and submission kernels: the base and kaggle-base environments, CPU only.
Every image is built for amd64 only. A release and its chart share one version number. In the published chart the defaults carry the digest of every image of the release: the app's and the broker's in image.app.digest and image.runtime.digest, the kernels' in the environment catalog catalog.environments. The same values come as a file of their own, values-X.Y.Z.yaml, attached to the release on GitHub: handy for copying the images into a mirror and for diffing releases in a GitOps repository. The chart in the source tree carries no digests and refuses to render without them.
The environment chosen for a class is a catalog entry: a name, an image by digest, a GPU flag. An environment of your own — other libraries, or a GPU — is built outside the cluster from the repository's kernel/ directory with KERNEL_ENV=<name>, pushed to your registry and added to catalog.environments: on Kubernetes the panel doesn't build environments. The GPU images, base-gpu (torch 2.14.1 and torchvision as CUDA 13.0 wheels, an image of about 9.9 GB) and kaggle-gpu on top of it (the kaggle-base set plus OpenCV and timm, about 2.8 GB more, 12.7 GB in all), are not published; build them the same way, the parent first. The nodes that run them on a card need an NVIDIA driver of 580 or newer. Every Pod of a GPU image, the metric's and package preparation's on the students' nodes included, has gpu.competitions.pullSeconds (900 seconds) to pull it onto a node that has never run it, before the job's own time starts; to spare the first class that wait, pull the GPU images onto those nodes in advance, for instance with a DaemonSet whose init containers are those images, each running true.
The registry mirror. global.imageRegistry replaces the registry in every image reference — the app's, the broker's, the catalog's kernels, the competition exporter and package proxy — and keeps the path and the digest. The value may carry a path too: a Harbor proxy cache of ghcr.io in a project called ghcr works as is, and harbor.corp.example/ghcr gives harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. So the mirror must hold the images under the same paths. Pull secrets go into global.imagePullSecrets: the app's and the broker's Pods get them, and every Pod the broker creates gets the first one.
Without internet access. On a machine that can reach ghcr.io:
# the release's chart
 helm pull oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z
 
 # every image it refers to, the catalog's kernels included
@@ -156,7 +156,7 @@ 
Installation
Your team creates the namespace with its Po
         print(host, port, "OPEN: check the network policies")
     except OSError as error:
         print(host, port, "closed:", error)
  • If you use competitions: a trial submission through to the end; leaderboard rows should appear by themselves, with no page reload.
  • Before the first group: a restore from a backup into a separate namespace.
    • 
-
    Chart values
    The main values and the variables they set. The full list, each value with an explanation, is the chart's values.yaml: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Other app settings from the Settings table of the VM guide go into config.extraEnv, or as keys of the Secret named in config.existingSecret.
    ValueVariable and what it setsDefault
    Address and ingress
    ingress.hostThe hostname; required while the Ingress is on.—
    config.publicUrlPUBLIC_URL — the external address links are built from.https://<ingress.host>
    ingress.className, ingress.annotationsThe controller class and annotations.nginx, the annotations under Network
    ingress.tls.secretName, ingress.tls.certManager.*The certificate's Secret, or a cert-manager issuer.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — whose X-Forwarded-For to believe.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — NAT and VPN addresses that per-address caps don't apply to.empty
    config.inbound.hstsHSTS — the app's Strict-Transport-Security header.true
    Images and the catalog
    global.imageRegistryA registry, optionally with a path, instead of ghcr.io for every image, the kernel catalog included.empty
    global.imagePullSecretsSecrets for pulling images; broker Pods get the first one (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestDigests of the app and broker images.the release's digests
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — the environment catalog: a name, an image by digest, GPU; several revisions of one name with exactly one current: true.the release's base and kaggle-base
    commonLabelsLabels on every chart object and every broker Pod (part of RUNTIME_POD_LABELS).—
    Secrets
    config.existingSecretAn existing Secret whose keys become the app's variables.—
    secrets.runtimeToken.*, secrets.roomSecret.*The broker token and the room secret: existingSecret and existingSecretKey, or value.generated once by the chart (helm CLI only)
    Storage
    persistence.accessModeThe volumes' access mode. If either volume is ReadWriteOnce, the broker gets RUNTIME_COLOCATE_WITH_APP=1: every Pod with a volume runs on the app's node.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeThe storage class and volume sizes; existingClaim attaches a PVC you have.the cluster default; 10Gi and 100Gi
    persistence.retainKeep the PVCs when the release or the ArgoCD application is deleted.true
    Rooms
    rooms.memoryRUNTIME_KERNEL_MEMORY — a room's memory unless the class sets its own.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — a room's memory ceiling. Set it explicitly: without it the broker takes its own node's memory minus 1 GiB.the broker node's memory − 1 GiB
    rooms.cpuRUNTIME_KERNEL_CPU — a room's cores unless the class sets its own.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — space for a room's /tmp and home directory.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — with none, rooms get no DNS either.DNS only
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — where room and personal-notebook Pods run; competition Pods don't get them.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — the priority class of every Pod the broker creates.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — labels and annotations on every broker Pod; the chart adds app.kubernetes.io/name, instance and part-of to the labels itself.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — with false, live rooms aren't resized and the Role has no pods/resize rule.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — the RuntimeClass of GPU rooms; empty means none.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — added for GPU rooms.—
    Network and outbound
    networkPolicy.ingressController.fromWho is admitted to the app's port 3000.ingress-nginx
    networkPolicy.dns.toWhere cluster DNS is.kube-dns in kube-system
    networkPolicy.kubeApiServer.*The API servers' addresses and ports for the broker; ciliumEntity for Cilium.any address on 443 and 6443
    networkPolicy.app.egressWhere the app may go outside: the model, a proxy, GitHub.nowhere
    networkPolicy.monitoring.fromWho scrapes /metrics.Prometheus in monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — the app's outbound proxy. Service names (.svc, single-label names) and private addresses bypass it by themselves.—
    outbound.extraCa.pem or .existingConfigMapNODE_EXTRA_CA_CERTS — the internal CA's certificates as PEM, for the app and package preparation.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — the PyPI mirror for competition entrants' own packages: https, no credentials.public PyPI
    dependencies.mirrorEgress, dependencies.publicEgressThe mirror's addresses and ports for the package proxy; the general rule for public addresses on 443.—; true
    Classes and the Oracle
    config.timezoneTZ — the time zone: day boundaries for daily quotas, dates.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — the interface language until the owner chooses one in the panel.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — the limit for one file and for all uploads to a room.50 and 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — the Oracle's model; for a model on the internal network, vllm, ollama or custom. The key is better kept in config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvOther app settings, as NAME: value.—
    Monitoring, backups, resources
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — /metrics behind a Bearer token, and a ServiceMonitor that uses the same token.off
    config.logFormatLOG_FORMAT — with json, one JSON object per line.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — how often to write a consistent database snapshot into /data/snapshots, and how many to keep.24 h and 7
    app.resources, runtime.resourcesRequests and limits of the app and the broker.see Resources and quotas
    
+
    Chart values
    The main values and the variables they set. The full list, each value with an explanation, is the chart's values.yaml: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Other app settings from the Settings table of the VM guide go into config.extraEnv, or as keys of the Secret named in config.existingSecret.
    ValueVariable and what it setsDefault
    Address and ingress
    ingress.hostThe hostname; required while the Ingress is on.—
    config.publicUrlPUBLIC_URL — the external address links are built from.https://<ingress.host>
    ingress.className, ingress.annotationsThe controller class and annotations.nginx, the annotations under Network
    ingress.tls.secretName, ingress.tls.certManager.*The certificate's Secret, or a cert-manager issuer.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — whose X-Forwarded-For to believe.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — NAT and VPN addresses that per-address caps don't apply to.empty
    config.inbound.hstsHSTS — the app's Strict-Transport-Security header.true
    Images and the catalog
    global.imageRegistryA registry, optionally with a path, instead of ghcr.io for every image, the kernel catalog included.empty
    global.imagePullSecretsSecrets for pulling images; broker Pods get the first one (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestDigests of the app and broker images.the release's digests
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — the environment catalog: a name, an image by digest, GPU; several revisions of one name with exactly one current: true.the release's base and kaggle-base
    commonLabelsLabels on every chart object and every broker Pod (part of RUNTIME_POD_LABELS).—
    Secrets
    config.existingSecretAn existing Secret whose keys become the app's variables.—
    secrets.runtimeToken.*, secrets.roomSecret.*The broker token and the room secret: existingSecret and existingSecretKey, or value.generated once by the chart (helm CLI only)
    Storage
    persistence.accessModeThe volumes' access mode. If either volume is ReadWriteOnce, the broker gets RUNTIME_COLOCATE_WITH_APP=1: every Pod with a volume runs on the app's node.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeThe storage class and volume sizes; existingClaim attaches a PVC you have.the cluster default; 10Gi and 100Gi
    persistence.retainKeep the PVCs when the release or the ArgoCD application is deleted.true
    Rooms
    rooms.memoryRUNTIME_KERNEL_MEMORY — a room's memory unless the class sets its own.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — a room's memory ceiling. Set it explicitly: without it the broker takes its own node's memory minus 1 GiB.the broker node's memory − 1 GiB
    rooms.cpuRUNTIME_KERNEL_CPU — a room's cores unless the class sets its own.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — space for a room's /tmp and home directory.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — with none, rooms get no DNS either.DNS only
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — where room, personal-notebook and competition Pods run: submissions, the metric and package preparation (not its package proxy). A GPU submission's Pod gets gpu.nodeSelector and gpu.tolerations on top, so its node must match both.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — the priority class of every Pod the broker creates.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — labels and annotations on every broker Pod; the chart adds app.kubernetes.io/name, instance and part-of to the labels itself.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — with false, live rooms aren't resized and the Role has no pods/resize rule.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — the RuntimeClass of GPU rooms and GPU submissions; empty means none.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — added for GPU rooms and GPU submissions.—
    gpu.competitions.resource, gpu.competitions.countRUNTIME_COMPETITION_GPU_RESOURCE, RUNTIME_COMPETITION_GPU_COUNT — what the notebook Pod of a competition with the GPU switch on requests: the device plugin's resource name, such as nvidia.com/gpu or nvidia.com/mig-1g.10gb, and how many units of it, 1 to 8. nvidia.com/gpu is what GPU rooms request too: with it, submissions take the rooms' cards; see GPU nodes. Empty: submissions get no GPU, and such a competition refuses uploads.empty, 1
    gpu.competitions.slotsRUNTIME_COMPETITION_GPU_SLOTS — how many GPU submissions the cluster runs at once for this installation, 1 to 256: the units set aside for submissions over count. The app's queue keeps that many at the broker and lets CPU work go past the rest; see GPU nodes.empty: one at a time
    gpu.competitions.shmMbRUNTIME_COMPETITION_GPU_SHM_MB — the ceiling of a GPU submission's in-memory /dev/shm, MiB, 0 to 65536; it is never more than half the submission's memory. 0: none, which MPS sharing needs.1024
    gpu.competitions.pullSecondsRUNTIME_COMPETITION_GPU_PULL_SECONDS — how long any Pod of a GPU catalog image may take to pull it onto its node before the job's own time starts, 0 to 3600 seconds: a submission, the metric and package preparation alike, with a GPU or without one.900
    Network and outbound
    networkPolicy.ingressController.fromWho is admitted to the app's port 3000.ingress-nginx
    networkPolicy.dns.toWhere cluster DNS is.kube-dns in kube-system
    networkPolicy.kubeApiServer.*The API servers' addresses and ports for the broker; ciliumEntity for Cilium.any address on 443 and 6443
    networkPolicy.app.egressWhere the app may go outside: the model, a proxy, GitHub.nowhere
    networkPolicy.monitoring.fromWho scrapes /metrics.Prometheus in monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — the app's outbound proxy. Service names (.svc, single-label names) and private addresses bypass it by themselves.—
    outbound.extraCa.pem or .existingConfigMapNODE_EXTRA_CA_CERTS — the internal CA's certificates as PEM, for the app and package preparation.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — the PyPI mirror for competition entrants' own packages: https, no credentials.public PyPI
    dependencies.mirrorEgress, dependencies.publicEgressThe mirror's addresses and ports for the package proxy; the general rule for public addresses on 443.—; true
    Classes and the Oracle
    config.timezoneTZ — the time zone: day boundaries for daily quotas, dates.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — the interface language until the owner chooses one in the panel.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — the limit for one file and for all uploads to a room.50 and 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — the Oracle's model; for a model on the internal network, vllm, ollama or custom. The key is better kept in config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvOther app settings, as NAME: value.—
    Monitoring, backups, resources
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — /metrics behind a Bearer token, and a ServiceMonitor that uses the same token.off
    config.logFormatLOG_FORMAT — with json, one JSON object per line.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — how often to write a consistent database snapshot into /data/snapshots, and how many to keep.24 h and 7
    app.resources, runtime.resourcesRequests and limits of the app and the broker.see Resources and quotas
    
 
    Resources and quotas
    The numbers come from the limits in the code and from the same reasoning as for one VM, turned here into Pod requests and limits. The main difference from a VM: a room's Pod reserves its memory and cores in full rather than sharing them with its neighbours.
    PodHow manyCPU: request / limitMemory: request / limitEphemeral storage
    App1250m / 2512Mi / 2Gi—
    Broker1100m / 1128Mi / 512Mi—
    Roomone per running room2 / 24Gi / 4Gi2Gi
    Personal notebooksone per room that uses themthe room's, or the class's own number, request equal to limit; about 250m per student is neededthe room's, or Memory per class; 1–1.5 GiB per student is needed2Gi
    Submissionone per slotthe submission's cores + 100m / + 500mthe submission's memory + 64Mi / + 256Mi128Mi / 320Mi
    Package preparationwhile it runs1100m / 1500m, proxy 100m / 500m2112Mi / 2304Mi, proxy 64Mi / 128Miup to 320Mi
    The minimum quota for one class at a time (a room of 2 cores and 4 GiB), in the ResourceQuota keys requests.cpu, requests.memory, limits.cpu, limits.memory and pods:
    ScenarioCPU requestsMemory requestsCPU limitsMemory limitsPods
    (a) 30 students, one shared notebook2.354.6 GiB56.5 GiB3
    (b) 30 students, each with a personal notebook (1.5 GiB and 0.25 core per student)9.8549.6 GiB12.551.5 GiB4
    (c) 30 students and a competition, 7 slots of 2 cores and 2 GiB17.0519.1 GiB22.522.25 GiB10
    (e) 100 people in a lecture room2.354.6 GiB56.5 GiB3
    A room keeps its Pod while anyone is in it and for 2 hours after everyone has left, so back-to-back classes overlap: for each further class within two hours, add another room. Competition slots are one queue for the whole installation. A quota for scenario (c) with headroom for the next class's room and for package preparation:
    apiVersion: v1
 kind: ResourceQuota
 metadata:
@@ -172,7 +172,7 @@ 
    Resources and quotas
    The numbers come from the limits in
     pods: "25"
     services: "100"
     persistentvolumeclaims: "2"
-    requests.storage: 150Gi
    • The API won't create a Pod over the quota, and the room or the submission gets a refusal. Keep the quota above the sum and the slot count within it: Executors at once in the Resources tab, or COMPETITION_SLOTS in config.extraEnv. The automatic count works from the cores and memory of the node the app runs on and knows nothing about your quota.
    • Services pile up: every room deleted for good leaves a headless Service without a ClusterIP, so give the Service count limit headroom.
    • A LimitRange with a per-container maximum must admit a room (2 cores and 4 GiB; a GPU room with its memory) and the largest submission you allow in competitions: 2 cores and 4 GiB by default.
    • With ReadWriteOnce the whole quota must fit on one node: for scenario (c), a node with at least 24 vCPU and 32 GiB free; for (b), at least 16 vCPU and 64 GiB.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core. This was measured on a VM, not in a cluster, but the process is the same: the chart's request of 512 MiB is twice the measured figure, and the limits of 2 cores and 2 GiB leave plenty of headroom.
    
+    requests.storage: 150Gi
    • The API won't create a Pod over the quota, and the room or the submission gets a refusal. Keep the quota above the sum and the slot count within it: Executors at once in the Resources tab, or COMPETITION_SLOTS in config.extraEnv. The automatic count works from the cores and memory of the node the app runs on and knows nothing about your quota. A GPU submission refused that way goes back to the queue and is retried instead. A quota that names the GPU resource, requests.nvidia.com/gpu, bounds GPU rooms and GPU submissions together: keep gpu.competitions.slots × gpu.competitions.count within it, less what GPU rooms hold.
    • Services pile up: every room deleted for good leaves a headless Service without a ClusterIP, so give the Service count limit headroom.
    • A LimitRange with a per-container maximum must admit a room (2 cores and 4 GiB; a GPU room with its memory) and the largest submission you allow in competitions: 2 cores and 4 GiB by default.
    • With ReadWriteOnce the whole quota must fit on one node: for scenario (c), a node with at least 24 vCPU and 32 GiB free; for (b), at least 16 vCPU and 64 GiB.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core. This was measured on a VM, not in a cluster, but the process is the same: the chart's request of 512 MiB is twice the measured figure, and the limits of 2 cores and 2 GiB leave plenty of headroom.
    
 
    Operations
    Upgrades. helm upgrade colloq oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z -n colloq -f colloq-values.yaml, or a new targetRevision in ArgoCD. The app and the broker update with the Recreate strategy: the old Pod stops before the new one starts, because SQLite has one writer. The break lasts seconds to a minute; a start with a database migration is given up to 10 minutes. Open rooms reconnect by themselves. Room Pods survive the upgrade: the broker leaves them alone when it stops, the new one adopts them, and Python variables stay. Placement settings — rooms.nodeSelector, rooms.tolerations, the priority class, the storage mode — go to a room's next Pod, while a running one stays where it is. Still, upgrade between classes.
    Kernel revisions. A room stays on the kernel image revision it started with and runs only that one. On helm upgrade the chart keeps the previous release's revisions in the catalog with current: false by itself (at most catalog.maxRetainedPerEnvironment, three by default, per environment): it reads them from the live colloq-catalog ConfigMap, and rooms started earlier go on with their Python. ArgoCD renders without cluster access and cannot see the previous catalog: there a room whose revision is gone moves to its environment's current revision on its next start, and the app logs a line about it. To keep previous revisions under ArgoCD too, list them in catalog.environments with current: false. Keep the older revisions' images in the mirror while rooms use them.
    The data version. If a new release raises the database schema version, the app first writes a snapshot, /data/snapshots/pre-schema-<from>-to-<to>-<time>.db; such snapshots are never removed automatically. An older release won't start on a database with a newer schema: the log says "colloq.db … was written by a newer Colloq", and the Pod goes into CrashLoopBackOff. So a rollback in that case is the older chart together with restoring the database: stop the app, put the snapshot in place of colloq.db from a maintenance Pod and delete colloq.db-wal and colloq.db-shm, then install the older chart version. Anything done after the upgrade is lost. COLLOQ_ALLOW_SCHEMA_DOWNGRADE=1 in config.extraEnv starts the older release on the newer database without a restore — only if you know the migration is harmless for it.
    kubectl -n colloq scale deploy/colloq-app --replicas=0
 kubectl -n colloq apply -f - <<'EOF'
 apiVersion: v1
@@ -205,5 +205,5 @@ 
    Operations
    Upgrades. helm upgr
 kubectl -n colloq exec colloq-maintenance -- sh -c \
   'cp /data/snapshots/pre-schema-….db /data/colloq.db && rm -f /data/colloq.db-wal /data/colloq.db-shm'
 kubectl -n colloq delete pod colloq-maintenance
-helm rollback colloq <revision before the upgrade> -n colloq
    Backups. Snapshot both PVCs with your own tools — Velero with CSI snapshots, or your storage's snapshots — and the namespace's Secrets unless they come from Vault, in which case Vault is their backup. A snapshot of a live SQLite volume is a crash image: the database usually recovers from it, but it is not something to rely on. So every config.dbSnapshotHours hours (24 in the chart) the app itself writes a consistent snapshot, /data/snapshots/colloq-<time>.db, and keeps the last config.dbSnapshotKeep (7): every volume snapshot holds a whole copy of the database no older than a day. The volumes are snapshotted separately, so room files and the database in a backup can be minutes apart. Copies of the colloq-data volume contain the teachers' sign-in keys, the setup token and the model key if it was entered in the panel: keep them as secrets. Test a restore into a separate namespace at least once a term. The colloq-host backup and restore commands are for the VM and are not used in a cluster.
    Monitoring. The liveness probe is /api/livez, readiness is /api/readyz: it checks only the database and the working files, so a broker or API hiccup doesn't take the site out of rotation, and kernel trouble shows where code is run and in /api/health. With metrics.enabled, /metrics serves Prometheus text to requests carrying Authorization: Bearer <token>; without the setting it answers 404. Useful for alerts: free space on the volumes (colloq_disk_free_bytes), the time of the last database snapshot (colloq_db_snapshot_last_timestamp_seconds), notebook save and kernel start failures (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Logs go to stdout and stderr; with config.logFormat: json each line is a JSON object with time, level, msg and context fields. The log contains room addresses, and a room address is a sign-in link, so access to the log equals access to the rooms. Staff can use GET /api/instance/operations: queue age, retries, notebook save failures, memory reservations, free disk.
    GPU nodes. A GPU room requests nvidia.com/gpu: 1 and the RuntimeClass from gpu.runtimeClassName (nvidia by default; empty means none); gpu.nodeSelector and gpu.tolerations pick the nodes. The driver, the device plugin or GPU Operator and the RuntimeClass itself are cluster-scoped objects your team provides. One card per room, not shared, unless your device plugin splits cards itself. Personal notebooks and competition submissions never get a GPU. Build the GPU environment's image yourself; see Images and the registry mirror.
    Node maintenance. Room Pods are bare Pods without a controller and with emptyDir: kubectl drain removes them only with --force --delete-emptydir-data, and the cluster autoscaler won't free a node that runs them until you add the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" through rooms.podAnnotations. A room brings a deleted Pod back at its next run: notebooks and files are there, Python variables are not. With ReadWriteOnce, maintenance on the app's node is a break for the whole installation.
    Uninstalling. helm uninstall removes the chart's objects except the PVCs — with persistence.retain: true they stay — and leaves the Pods and Services the broker created: they don't belong to Helm. Delete them by label: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
-
    Honest limits
    • One app replica. The database is SQLite with one writer: there is no high availability, and an upgrade or a node failure interrupts work while the Pod restarts. To grow, run separate installations, one per namespace — for example one per department.
    • SSO only through a proxy. Colloq has no SSO, LDAP or second-factor sign-in of its own: a proxy that signs a JWT provides it — see Sign-in through your SSO. Without such a proxy the owner and teachers sign in with personal links, and the panel can be closed off at the ingress controller — for example oauth2-proxy with the corporate IdP through ingress-nginx's external authentication annotations, on a separate Ingress for the paths /admin and /api/admin. Competition entrants sign in with their own key even behind the proxy.
    • A room link is a shared link. Whoever has it can join and, by default, run code; it can't be revoked. Rooms don't suit individually graded or confidential work: participants share the kernel, the files and the Linux user.
    • ReadWriteOnce means one node. The whole installation fits on one node, and its maintenance is a break for everyone.
    • Students' code has no internet. pip install in a cell doesn't work; libraries come only in the catalog's kernel images, built outside the cluster.
    • No per-room disk quota: the colloq-workspace volume is shared.
    • Network policies are not all of the isolation. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node, and Pods share the node's Linux kernel.
    • amd64 only.
    • Kernels on Debian 12. A scanner finds critical findings in them that Debian has not fixed; see “Images and the registry mirror”.
    • A new path. The chart is new in this release: test the installation in a non-production namespace before running classes in it.
    
+helm rollback colloq <revision before the upgrade> -n colloq
    Backups. Snapshot both PVCs with your own tools — Velero with CSI snapshots, or your storage's snapshots — and the namespace's Secrets unless they come from Vault, in which case Vault is their backup. A snapshot of a live SQLite volume is a crash image: the database usually recovers from it, but it is not something to rely on. So every config.dbSnapshotHours hours (24 in the chart) the app itself writes a consistent snapshot, /data/snapshots/colloq-<time>.db, and keeps the last config.dbSnapshotKeep (7): every volume snapshot holds a whole copy of the database no older than a day. The volumes are snapshotted separately, so room files and the database in a backup can be minutes apart. Copies of the colloq-data volume contain the teachers' sign-in keys, the setup token and the model key if it was entered in the panel: keep them as secrets. Test a restore into a separate namespace at least once a term. The colloq-host backup and restore commands are for the VM and are not used in a cluster.
    Monitoring. The liveness probe is /api/livez, readiness is /api/readyz: it checks only the database and the working files, so a broker or API hiccup doesn't take the site out of rotation, and kernel trouble shows where code is run and in /api/health. With metrics.enabled, /metrics serves Prometheus text to requests carrying Authorization: Bearer <token>; without the setting it answers 404. Useful for alerts: free space on the volumes (colloq_disk_free_bytes), the time of the last database snapshot (colloq_db_snapshot_last_timestamp_seconds), notebook save and kernel start failures (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Logs go to stdout and stderr; with config.logFormat: json each line is a JSON object with time, level, msg and context fields. The log contains room addresses, and a room address is a sign-in link, so access to the log equals access to the rooms. Staff can use GET /api/instance/operations: queue age, retries, notebook save failures, memory reservations, free disk.
    GPU nodes. A GPU room requests nvidia.com/gpu: 1 and the RuntimeClass from gpu.runtimeClassName (nvidia by default; empty means none); gpu.nodeSelector and gpu.tolerations pick the nodes. The driver, the device plugin or GPU Operator and the RuntimeClass itself are cluster-scoped objects your team provides. The shipped GPU environments need a driver of 580 or newer: base-gpu and kaggle-gpu carry torch 2.14.1 as CUDA 13.0 wheels, which import on an older driver but never see the card. One card per room, not shared, unless your device plugin splits cards itself. Personal notebooks never get a GPU. A competition submission gets one only when its competition's GPU switch is on and gpu.competitions.resource is set: its notebook Pod, never the metric's, requests gpu.competitions.count units of that resource, with the rooms' nodeSelector and tolerations and, on top of them, the GPU rooms' RuntimeClass, nodeSelector and tolerations, and an in-memory /dev/shm of half its memory, at most gpu.competitions.shmMb (1024 MiB). What one unit is — a whole card, a time-sliced or MPS replica, a MIG slice — is your device plugin's configuration, and Colloq caps no video memory here itself. With MPS sharing, set gpu.competitions.shmMb: 0, because the device plugin mounts its MPS daemon's shared memory at /dev/shm, a volume of ours there would hide it, and CUDA would not start in the submission; and keep GPU rooms off MPS nodes, which renamed replicas (below) do, since a room keeps its own 1 GiB /dev/shm. Without the value, a competition with the switch on refuses uploads and says why. How many GPU submissions go to the cluster at once is gpu.competitions.slots, one while it is unset: the app's queue keeps that many at the broker, and CPU submissions go past the rest; once the cluster has sent one back for want of a unit, the queue takes no GPU submission until that one's retry is due or one of the running GPU submissions ends. A GPU submission the scheduler cannot place, or whose Pod the API server refuses — a ResourceQuota used up, a RuntimeClass the cluster lacks, an admission policy — goes back to the queue and is retried a few seconds to a minute later, and its teacher sees the cause. With nvidia.com/gpu, the resource GPU rooms request, submissions take units from the rooms' cards: a submission holds its unit for up to the competition's time limit, and a GPU room that finds no card free is refused after 20 seconds. Where classes use GPU rooms, give submissions a resource of their own, so the two never take the same cards: time-sliced replicas renamed to nvidia.com/gpu.shared (renameByDefault: true) on nodes labelled for that device-plugin configuration, or a MIG profile such as nvidia.com/mig-1g.10gb. Build the GPU environments' images yourself; see Images and the registry mirror.
    Node maintenance. Room Pods are bare Pods without a controller and with emptyDir: kubectl drain removes them only with --force --delete-emptydir-data, and the cluster autoscaler won't free a node that runs them until you add the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" through rooms.podAnnotations. A room brings a deleted Pod back at its next run: notebooks and files are there, Python variables are not. With ReadWriteOnce, maintenance on the app's node is a break for the whole installation.
    Uninstalling. helm uninstall removes the chart's objects except the PVCs — with persistence.retain: true they stay — and leaves the Pods and Services the broker created: they don't belong to Helm. Delete them by label: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
+
    Honest limits
    • One app replica. The database is SQLite with one writer: there is no high availability, and an upgrade or a node failure interrupts work while the Pod restarts. To grow, run separate installations, one per namespace — for example one per department.
    • SSO only through a proxy. Colloq has no SSO, LDAP or second-factor sign-in of its own: a proxy that signs a JWT provides it — see Sign-in through your SSO. Without such a proxy the owner and teachers sign in with personal links, and the panel can be closed off at the ingress controller — for example oauth2-proxy with the corporate IdP through ingress-nginx's external authentication annotations, on a separate Ingress for the paths /admin and /api/admin. Competition entrants sign in with their own key even behind the proxy.
    • A room link is a shared link. Whoever has it can join and, by default, run code; it can't be revoked. Rooms don't suit individually graded or confidential work: participants share the kernel, the files and the Linux user.
    • ReadWriteOnce means one node. The whole installation fits on one node, and its maintenance is a break for everyone.
    • Students' code has no internet. pip install in a cell doesn't work; libraries come only in the catalog's kernel images, built outside the cluster.
    • No per-room disk quota: the colloq-workspace volume is shared.
    • Network policies are not all of the isolation. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node, and Pods share the node's Linux kernel.
    • A GPU submission's share is the device plugin's. Colloq neither caps nor watches video memory on Kubernetes: a MIG slice has memory of its own, while time-sliced replicas share a card's memory and compute with nothing between them, and one submission can take the memory the others need.
    • amd64 only.
    • Kernels on Debian 12. A scanner finds critical findings in them that Debian has not fixed; see “Images and the registry mirror”.
    • A new path. The chart is new in this release: test the installation in a non-production namespace before running classes in it.
    
diff --git a/docs/pages/en/server.html b/docs/pages/en/server.html
index ab45a069..158e611f 100644
--- a/docs/pages/en/server.html
+++ b/docs/pages/en/server.html
@@ -1,7 +1,7 @@
 
    There are two ways to run Colloq at a university; pick the one that fits your infrastructure.
    A / YOU HAVE A KUBERNETES CLUSTER
    In your Kubernetes
    A Helm chart in one namespace: no cluster-admin, restricted Pod Security, your own registry, storage, ingress and secrets from Vault. For organisations whose platform team runs the cluster.
    B / ONE MACHINE
    On one VM
    The ready-made image and colloq-host on a dedicated Linux machine behind the university's reverse proxy. The rest of this page is about that.
    
 
    In short for IT
    What it isColloq is a collaborative Jupyter classroom: a web app plus a separate container with a Python kernel for each room. Open source, MIT license.
    MachineOne dedicated x86-64 VM with Ubuntu 22.04 or 24.04 and Docker Engine 28 or newer. The app controls Docker, which is root on the VM, so run nothing else on it.
    ResourcesStart with 16 vCPU, 64 GB RAM, 250 GB NVMe, no GPU. That covers a group of 30 with personal notebooks or with a competition, and a lecture for 100; the reasoning is under Machine and sizing.
    InboundOnly HTTPS from users' browsers to your reverse proxy; the proxy reaches the app at 127.0.0.1:3000. If users are on the campus network or VPN, nothing is needed from the internet.
    OutboundNothing at run time for the core features: no telemetry, no update checks, no CDN. Installing and updating need ghcr.io — or your registry mirror, or carrying the images over as a file. Optional: an AI model (OpenAI or your own on the internal network), notebook import from GitHub, PyPI or an internal mirror for competition packages, internet access for students' code.
    DataAll on this VM under /workspace/colloq: the SQLite database, room files, competition data, backups. Settings are in /etc/colloq/colloq.env.
    Backupscolloq-host backup by hand, or colloq-host backup-timer on for one every night. Copying the backups off the machine is up to your own tools.
    
 
    How it fits together
    Students' and teachers' browsersUniversity reverse proxy, TLSColloq container
    Room A kernelRoom B kernelCompetition submission
    The university's reverse proxy terminates HTTPS and forwards requests to the app. The app is one container, colloq, from the image ghcr.io/colloq-edu/colloq-server: a Node.js server, the built web interface and an SQLite database. Its port 3000 is published on 127.0.0.1 of this VM only. The container is started and updated by colloq-host, a small script on the host that is installed from the same image.
    For each room the app starts a kernel container of its own on the same Docker: its own memory, CPU and process limits, only that room's folder mounted, uid 1000, no Linux capabilities. Students' personal notebooks run in a second container per class; a competition submission runs in two throwaway containers with no network. The app and the room kernels share the Docker network colloq.
    The state directory is /workspace/colloq: data/ (the database and keys), workspace/<room>/ (room files), environments/ (environments created in the panel) and backups/. It is mounted into the container at the same path. Settings live in /etc/colloq/colloq.env (mode 0600) and the chosen image in /etc/colloq/image. Mount a separate disk or partition at /workspace/colloq, so that filling it up doesn't take the system down with it.
    
-
    Machine and sizing
    Start here: 16 vCPU, 64 GB RAM, 250 GB NVMe, x86-64 (the image is built for amd64 only), Ubuntu 22.04 or 24.04, Docker Engine 28 or newer, no GPU. The clock must be synchronised over NTP (from an internal server if there is no internet): sign-in links, cookies and certificates depend on the time. Docker must manage iptables itself: the room network block goes into its DOCKER-USER chain, so "iptables": false in daemon.json and Docker 29's nftables backend won't do. The machine's memory and CPU go to room kernels and submissions; the numbers for those below are derived from the limits in the code. The app itself is light, and that was measured.
    ScenariovCPURAMDisk
    (a) 30 students, one shared notebook48 GB60 GB
    (b) 30 students, each with a personal notebook1664 GB (48 GB at least)100 GB
    (c) 30 students and a competition, 2-core / 2 GB submissions16 (auto gives 7 slots)32 GB100 GB
    (d) three (c) classes in the same hour32 (about 15 slots)64 GB; 160–192 GB if all three use personal notebooks200 GB
    (e) 100 people in a lecture room4 (8 if students run cells)8 GB60 GB
    The starting point covers (a), (b), (c) and (e), and (d) with a slower submission queue. Assumptions: pandas and scikit-learn at the default settings, and a submission that runs for about two minutes.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core.
    Work out RAM like this: about 3 GB for the system and the app + room limits (4 GB per room by default) + personal-notebook container limits + slots × (submission memory + 0.25 GB) + 1.6 GB while packages are being prepared. Docker reserves nothing, so Colloq itself refuses to start a kernel whose limit, together with the limits of the containers already running, doesn't fit in the machine's memory minus 1 GB. CPU is shared, not reserved: a room is capped at 2 cores.
    • Personal notebooks: about 1–1.5 GB and 0.25 core per student. All personal notebooks of a class live in one container, which by default gets as much as the room — 4 GB for everyone together. For a group, raise Memory per class in the panel: Resources → Students' personal notebooks (or in a particular class's rules).
    • Competitions: how many submissions run at once (Executors at once in the Resources tab) is worked out by default: the smaller of (cores − 2) / submission cores and (memory − 2 GB) / (submission memory + 0.25 GB), and never more than 32. Thirty two-minute submissions take about 9 minutes on 7 slots and about 20 minutes on 3 slots (an 8 vCPU / 16 GB machine).
    • A room keeps its memory while anyone is in it and for 2 hours after everyone has left: the end of a class doesn't stop its kernel. Back-to-back classes therefore overlap.
    • Disk: the kaggle-base image takes about 1 GB and base-gpu about 9 GB; after that, room files, competition submissions and backups grow. Class pages keep their own copies of the published notebooks and files in data/page-files/: about 0.3–0.8 GB per 33-class course, with a file used on several pages stored once. There is no per-room quota; see monitoring.
    • A GPU is optional. Only GPU environments (base-gpu) need one: a card, 16 GB RAM and about 10 GB of disk for each GPU room running at the same time, plus the NVIDIA driver and the NVIDIA Container Toolkit (colloq-host installs the toolkit itself through apt). Personal notebooks and competition submissions never get a GPU.
    The Resources tab has a memory bar: running rooms, personal notebooks and competition slots against the machine's memory minus 1 GB; when memory runs short, the bar says so. It also shows free disk on the data partition, with a warning under 15 % or 10 GB free; it does not show GPUs.
    
+
    Machine and sizing
    Start here: 16 vCPU, 64 GB RAM, 250 GB NVMe, x86-64 (the image is built for amd64 only), Ubuntu 22.04 or 24.04, Docker Engine 28 or newer, no GPU. The clock must be synchronised over NTP (from an internal server if there is no internet): sign-in links, cookies and certificates depend on the time. Docker must manage iptables itself: the room network block goes into its DOCKER-USER chain, so "iptables": false in daemon.json and Docker 29's nftables backend won't do. The machine's memory and CPU go to room kernels and submissions; the numbers for those below are derived from the limits in the code. The app itself is light, and that was measured.
    ScenariovCPURAMDisk
    (a) 30 students, one shared notebook48 GB60 GB
    (b) 30 students, each with a personal notebook1664 GB (48 GB at least)100 GB
    (c) 30 students and a competition, 2-core / 2 GB submissions16 (auto gives 7 slots)32 GB100 GB
    (d) three (c) classes in the same hour32 (about 15 slots)64 GB; 160–192 GB if all three use personal notebooks200 GB
    (e) 100 people in a lecture room4 (8 if students run cells)8 GB60 GB
    The starting point covers (a), (b), (c) and (e), and (d) with a slower submission queue. Assumptions: pandas and scikit-learn at the default settings, and a submission that runs for about two minutes.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core.
    Work out RAM like this: about 3 GB for the system and the app + room limits (4 GB per room by default) + personal-notebook container limits + slots × (submission memory + 0.25 GB) + 1.6 GB while packages are being prepared. Docker reserves nothing, so Colloq itself refuses to start a kernel whose limit, together with the limits of the containers already running, doesn't fit in the machine's memory minus 1 GB. CPU is shared, not reserved: a room is capped at 2 cores.
    • Personal notebooks: about 1–1.5 GB and 0.25 core per student. All personal notebooks of a class live in one container, which by default gets as much as the room — 4 GB for everyone together. For a group, raise Memory per class in the panel: Resources → Students' personal notebooks (or in a particular class's rules).
    • Competitions: how many submissions run at once (Executors at once in the Resources tab) is worked out by default: the smaller of (cores − 2) / submission cores and (memory − 2 GB) / (submission memory + 0.25 GB), and never more than 32. Thirty two-minute submissions take about 9 minutes on 7 slots and about 20 minutes on 3 slots (an 8 vCPU / 16 GB machine).
    • A room keeps its memory while anyone is in it and for 2 hours after everyone has left: the end of a class doesn't stop its kernel. Back-to-back classes therefore overlap.
    • Disk: the kaggle-base image takes about 1 GB, base-gpu about 9.9 GB, and kaggle-gpu about 2.8 GB more on top of base-gpu, 12.7 GB in all; after that, room files, competition submissions and backups grow. Class pages keep their own copies of the published notebooks and files in data/page-files/: about 0.3–0.8 GB per 33-class course, with a file used on several pages stored once. There is no per-room quota; see monitoring.
    • A GPU is optional. Only GPU environments (base-gpu, kaggle-gpu) need one: a card and 16 GB RAM for each GPU room running at the same time, plus an NVIDIA driver of 580 or newer and the NVIDIA Container Toolkit (colloq-host installs the toolkit itself through apt, never the driver). Both environments carry torch 2.14.1 as CUDA 13.0 wheels: on an older driver PyTorch imports but never sees the card, and nvidia-smi shows the driver's version in its header. Personal notebooks never get a GPU. Competition submissions get a share of a card only when the owner gives cards to submissions in the Resources tab and a competition turns its GPU switch on; a card given to submissions no longer serves rooms. See a card for submissions.
    The Resources tab has a memory bar: running rooms, personal notebooks and competition slots against the machine's memory minus 1 GB; when memory runs short, the bar says so. It also shows free disk on the data partition, with a warning under 15 % or 10 GB free. The bar does not count video memory; the cards, with their memory and whom each one serves, are listed in the GPUs for submissions block of the same tab. Their names and memory are read when the server starts and every ten minutes after, through a throwaway container of a kernel image that is given every card, since the server's own container has no nvidia-smi; until a kernel image exists, and always for a MIG instance, a card's size stays unknown.
    
 
    Inbound and outbound connections
    Inbound connections:
    From and toPortPurpose
    Users' browsers → reverse proxy443/TCP; 80 only to redirect to HTTPSThe whole app: pages, API, WebSocket, server-sent events.
    Reverse proxy → app127.0.0.1:3000 if the proxy is on this VM; otherwise the VM's internal address, port 3000 (COLLOQ_BIND)Plain HTTP inside the machine or a trusted segment.
    Administrators → VM22/TCPSSH and colloq-host; from the IT network only.
    Internet → VM—Not needed if users are on the campus network or VPN. Ports 80/443 from the internet are needed only if the proxy gets its own Let's Encrypt certificate.
    The kernels' Jupyter and the other internal ports are not published: they live inside the Docker network. Nothing calls out: the app serves its fonts, PDF viewer and plotly itself, and browsers only ever reach the Colloq address.
    Outbound connections:
    DestinationPurpose and whenRequiredCan it point elsewhere
    ghcr.io, pkg-containers.githubusercontent.comThe app image colloq-server and the kernel images colloq-kernel: installing and updatingYes, unless you carry the images over as a fileYes: a registry mirror in COLLOQ_IMAGE and KERNEL_IMAGE_REPO; without a network, a file
    get.docker.com, download.docker.comInstalling Docker if the machine has noneNo, if Docker is already installedInstall Docker from your own repository
    Docker Hub (registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com), deb.debian.org, pypi.org, files.pythonhosted.orgBuilding a kernel image on the machine: an environment from the panel, or a published image that couldn't be pulledNoPartly: --index-url lines in the environment's package list; ready kernel images from a registry
    nvidia.github.ioThe NVIDIA Container Toolkit, on the first install on a GPU machineGPU onlyInstall the package in advance
    The AI model, api.openai.com by defaultThe Oracle, when someone asks itNoYes: OPENAI_BASE_URL or the panel — Ollama, vLLM or another OpenAI-compatible address on the internal network
    api.github.com, raw.githubusercontent.comImporting a notebook from GitHub in the panel, on requestNoNo, github.com only
    pypi.org, files.pythonhosted.orgCompetition entrants' own packages, when a set is preparedNoYes: DEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS
    The internet from room kernelspip install, datasets and APIs in students' code, during classNoCan be cut off: COLLOQ_ROOM_NETWORK=none, see below
    
 
    Network for students' code
    • Competition submissions run with no network at all.
    • Rooms and personal notebooks reach the internet by default (pip install, datasets, APIs), but private addresses are closed to them: 10/8, 172.16/12, 192.168/16, 100.64/10, 169.254/16 with the cloud metadata, 127/8, multicast, the VM itself and neighbouring rooms. A connection there is refused at once; only DNS on port 53 stays open to any address. The app installs the rules itself, in iptables chains of its own, COLLOQ-ROOMS-*, reached from DOCKER-USER and INPUT. If they can't be installed, rooms don't start.
    • The university's public ranges. Students' code goes out from the VM's address, so services that trust university addresses (library subscriptions, internal sites on public addresses) will take it for one of their own. List those ranges: KERNEL_BLOCKED_CIDRS=192.0.2.0/24,198.51.100.0/24.
    • COLLOQ_ROOM_NETWORK=none gives room and personal-notebook kernels no outbound network at all; only the Colloq server reaches them. pip install in a cell then doesn't work: put the libraries you need into an environment in advance (panel → Environments).
    • COLLOQ_ROOM_NETWORK=open lifts the private-address block; use it only for trusted groups that need the local network.
    
 
    Outbound proxy, internal CA and PyPI mirror
    If the way out is only through a proxy, set HTTPS_PROXY, HTTP_PROXY and NO_PROXY (lowercase works too). All of the server's outbound HTTP — to the AI model and to GitHub — goes through the proxy, and pip gets the same values when it prepares packages. Private addresses and Colloq's own internal names (kernels, containers) bypass the proxy by themselves; list internal domains in NO_PROXY, such as your model's and your PyPI mirror's. NODE_EXTRA_CA_CERTS is the path to a PEM file with your internal CA's certificates on the host: colloq-host mounts it into the container at the same path, and pip gets it too. Put the same file into the system store as well: Docker pulls the images itself and needs both this CA and a proxy of its own.
    cp campus-ca.pem /usr/local/share/ca-certificates/campus-ca.crt
@@ -96,7 +96,7 @@ 
    Installation
    Everything below runs as root on a prepared
 colloq-host link
 colloq-host status
 colloq-host backup-timer on
    doctor changes nothing: one PASS, WARN or FAIL line per check — architecture, the Docker version and its firewall, memory and disk, port 3000, DNS, clock synchronisation, access to ghcr.io, PUBLIC_URL and trusted proxies. Fix every FAIL line before the first class. up installs what is missing, pulls the image, writes the settings into /etc/colloq/colloq.env and starts the colloq container. The default kernel image (base) is pulled from ghcr.io/colloq-edu/colloq-kernel, or built on the machine if that fails; this takes a few minutes in the background, and the panel works in the meantime. If the proxy is on another machine, add COLLOQ_BIND and TRUSTED_PROXIES to up; see Client addresses.
    colloq-host link prints the address and, while the server has no owner, the setup link …/admin/t/…. It is the key to the server: don't paste it into a group chat. Open it and enter your name and email, and you are the owner. Add teachers in the panel, under Who can teach: each gets a personal sign-in link.
    colloq-host status shows /api/health, the kernel images and the room containers; the server is ready for a class when the answer has "ok":true. Before the first group, create two rooms, open them from two devices on the students' network, run a cell, upload a file, then test a backup and a restore on a spare VM.
    
-
    Settings
    colloq-host up writes the variables from its environment into /etc/colloq/colloq.env (mode 0600): a non-empty value replaces the previous one, and a missing one erases nothing. It also remembers its own settings there — COLLOQ_HOME, COLLOQ_BIND — so that a later update acts the same way. To remove a line, edit the file. Running colloq-host up again applies the changes: the app container is recreated only if the settings or the image changed, and room kernels keep running. The owner can also change the Oracle, room resources and competition slots in the panel, without a restart; what the panel saved wins over the file.
    VariableWhat it setsDefault
    Address and proxy
    COLLOQ_TUNNELnone: your proxy provides the address. The other modes (relay, cloudflare, direct) are for rentals and for publishing without a proxy of your own.auto; off Vast without PUBLIC_URL, none with a warning
    PUBLIC_URLThe external address links are built from.—
    TRUSTED_PROXIESProxy addresses whose X-Forwarded-For is accepted: IPs, subnets, loopback, private, none.loopback; with the port on loopback, colloq-host adds the gateway of the colloq network
    SHARED_ADDRESSESNAT subnets that per-address caps don't apply to.empty
    HSTS0: don't send Strict-Transport-Security.on
    TRUST_CF_CONNECTING_IP1: accept CF-Connecting-IP; only for Colloq's own Cloudflare tunnels, never behind your proxy.off
    COLLOQ_BINDWhere the host publishes port 3000; needed when the proxy is on another machine. Read by colloq-host only.127.0.0.1
    Outbound connections
    HTTPS_PROXY, HTTP_PROXY, NO_PROXYThe outbound proxy for the AI model, GitHub import and pip.—
    NODE_EXTRA_CA_CERTSThe internal CA's PEM file (a path inside the container).—
    DEPENDENCY_INDEX_URLThe index for competition entrants' own packages.https://pypi.org/simple
    DEPENDENCY_FILES_HOSTSHosts that serve the package files, comma-separated.—
    COLLOQ_ROOM_NETWORKnone: rooms get no network; open: no private-address block.empty: internet yes, private addresses closed
    KERNEL_BLOCKED_CIDRSExtra subnets closed to rooms.—
    COLLOQ_HELPER_IMAGEThe image of the helper container that installs the iptables rules.the server's own image
    COLLOQ_OFFLINE1: download nothing; images come through colloq-host load, which sets it.off
    Images and backups
    COLLOQ_HOMEThe state directory; set it before the first up. Read by colloq-host only./workspace/colloq
    COLLOQ_IMAGEThe image of the first up; after that colloq-host update chooses the image (/etc/colloq/image).—
    KERNEL_IMAGE_REPOWhere ready kernel images come from: <repo>:v<version>-<environment>; none builds every environment on the machine.ghcr.io/colloq-edu/colloq-kernel
    COLLOQ_REGISTRY_USER, COLLOQ_REGISTRY_TOKENSign-in to a private registry (a read-only token); read by colloq-host only.—
    BACKUP_KEEPHow many recent backups to keep; 0 keeps them all.14
    BACKUP_AGE_RECIPIENTAn age public key: backups are encrypted for it. Restore with BACKUP_AGE_IDENTITY, the path to the private key.—
    COLLOQ_ALLOW_SCHEMA_DOWNGRADE1 opens a database written by a newer version. Without it an older image refuses such a database; the ordinary rollback is restore --image with the backup taken before the update.—
    COLLOQ_BACKUP_ATThe time of the daily backup; read by backup-timer on.03:30
    COLLOQ_BACKUP_BEFORE_UPDATE0: no backup before update; read by update.1
    COLLOQ_FREEZE_UPDATES1: turn off automatic system updates and hold the NVIDIA packages. Only the Vast on-start sets it; don't set it on a university server.off
    Classes
    INSTITUTIONThe name shown next to the logo.—
    UI_LANGUAGEThe interface language (ru, en) until the owner chooses one in the panel.ru
    TZThe time zone: day boundaries for daily quotas, dates.Europe/Moscow
    ADMIN_EMAILThe email the owner form suggests in advance.—
    MAX_UPLOAD_MBThe limit for one file uploaded to a room.50
    MAX_SESSION_MBThe limit for all uploads to one room; not a disk quota.1024
    KERNEL_MEMA room kernel's memory; KERNEL_MEM_<ENVIRONMENT> for one environment.4g, 16g for GPU environments
    KERNEL_CPUSCPU cores per room.2
    KERNEL_OWN_MAX, KERNEL_OWN_PIDS, KERNEL_OWN_IDLE_MINThe personal-notebook container: how many kernels it holds, its process ceiling, idle minutes before a kernel is stopped.60, 2048, 30
    KERNEL_ROOM_IDLE_MINMinutes an empty class (nobody inside, nothing computing) keeps its kernels; 0 keeps them. Also set on the Resources tab.120
    KERNEL_ENV, KERNEL_PRELOADThe default environment; which environments to prepare at start.base; same as KERNEL_ENV
    KERNEL_GPUSCards for GPU rooms.all detected
    OPEN_SEMINAR_CREATIONAllow creating rooms through the API without a teacher's sign-in.false
    SESSION_SECRETThe link-signing key; a new value invalidates every link handed out.generated, data/session-secret
    Oracle
    AI_PROVIDERopenai, ollama, vllm, openrouter or custom.openai
    OPENAI_API_KEYThe model key; without it the Oracle is off (Ollama and vLLM work without a key).—
    OPENAI_BASE_URLThe model's address.https://api.openai.com/v1
    OPENAI_MODELThe model.gpt-4o-mini
    Sign-in through a proxy (SSO)
    AUTH_JWT_JWKS_URL, AUTH_JWT_AUDIENCE, AUTH_JWT_ISSUER and the other AUTH_JWT_*Sign-in through Teleport or another proxy that signs a JWT: teachers are recognised by email, students join under their own name. How it works and which token fields are read is in the Kubernetes guide; on a machine the same variables go to colloq-host up. When Teleport publishes the app from this same machine: COLLOQ_TUNNEL=none, and PUBLIC_URL is the app's address in Teleport.off
    
+
    Settings
    colloq-host up writes the variables from its environment into /etc/colloq/colloq.env (mode 0600): a non-empty value replaces the previous one, and a missing one erases nothing. It also remembers its own settings there — COLLOQ_HOME, COLLOQ_BIND — so that a later update acts the same way. To remove a line, edit the file. Running colloq-host up again applies the changes: the app container is recreated only if the settings or the image changed, and room kernels keep running. The owner can also change the Oracle, room resources and competition slots in the panel, without a restart; what the panel saved wins over the file.
    VariableWhat it setsDefault
    Address and proxy
    COLLOQ_TUNNELnone: your proxy provides the address. The other modes (relay, cloudflare, direct) are for rentals and for publishing without a proxy of your own.auto; off Vast without PUBLIC_URL, none with a warning
    PUBLIC_URLThe external address links are built from.—
    TRUSTED_PROXIESProxy addresses whose X-Forwarded-For is accepted: IPs, subnets, loopback, private, none.loopback; with the port on loopback, colloq-host adds the gateway of the colloq network
    SHARED_ADDRESSESNAT subnets that per-address caps don't apply to.empty
    HSTS0: don't send Strict-Transport-Security.on
    TRUST_CF_CONNECTING_IP1: accept CF-Connecting-IP; only for Colloq's own Cloudflare tunnels, never behind your proxy.off
    COLLOQ_BINDWhere the host publishes port 3000; needed when the proxy is on another machine. Read by colloq-host only.127.0.0.1
    Outbound connections
    HTTPS_PROXY, HTTP_PROXY, NO_PROXYThe outbound proxy for the AI model, GitHub import and pip.—
    NODE_EXTRA_CA_CERTSThe internal CA's PEM file (a path inside the container).—
    DEPENDENCY_INDEX_URLThe index for competition entrants' own packages.https://pypi.org/simple
    DEPENDENCY_FILES_HOSTSHosts that serve the package files, comma-separated.—
    COLLOQ_ROOM_NETWORKnone: rooms get no network; open: no private-address block.empty: internet yes, private addresses closed
    KERNEL_BLOCKED_CIDRSExtra subnets closed to rooms.—
    COLLOQ_HELPER_IMAGEThe image of the helper container that installs the iptables rules.the server's own image
    COLLOQ_OFFLINE1: download nothing; images come through colloq-host load, which sets it.off
    Images and backups
    COLLOQ_HOMEThe state directory; set it before the first up. Read by colloq-host only./workspace/colloq
    COLLOQ_IMAGEThe image of the first up; after that colloq-host update chooses the image (/etc/colloq/image).—
    KERNEL_IMAGE_REPOWhere ready kernel images come from: <repo>:v<version>-<environment>; none builds every environment on the machine.ghcr.io/colloq-edu/colloq-kernel
    COLLOQ_REGISTRY_USER, COLLOQ_REGISTRY_TOKENSign-in to a private registry (a read-only token); read by colloq-host only.—
    BACKUP_KEEPHow many recent backups to keep; 0 keeps them all.14
    BACKUP_AGE_RECIPIENTAn age public key: backups are encrypted for it. Restore with BACKUP_AGE_IDENTITY, the path to the private key.—
    COLLOQ_ALLOW_SCHEMA_DOWNGRADE1 opens a database written by a newer version. Without it an older image refuses such a database; the ordinary rollback is restore --image with the backup taken before the update.—
    COLLOQ_BACKUP_ATThe time of the daily backup; read by backup-timer on.03:30
    COLLOQ_BACKUP_BEFORE_UPDATE0: no backup before update; read by update.1
    COLLOQ_FREEZE_UPDATES1: turn off automatic system updates and hold the NVIDIA packages. Only the Vast on-start sets it; don't set it on a university server.off
    Classes
    INSTITUTIONThe name shown next to the logo.—
    UI_LANGUAGEThe interface language (ru, en) until the owner chooses one in the panel.ru
    TZThe time zone: day boundaries for daily quotas, dates.Europe/Moscow
    ADMIN_EMAILThe email the owner form suggests in advance.—
    MAX_UPLOAD_MBThe limit for one file uploaded to a room.50
    MAX_SESSION_MBThe limit for all uploads to one room; not a disk quota.1024
    KERNEL_MEMA room kernel's memory; KERNEL_MEM_<ENVIRONMENT> for one environment.4g, 16g for GPU environments
    KERNEL_CPUSCPU cores per room.2
    KERNEL_OWN_MAX, KERNEL_OWN_PIDS, KERNEL_OWN_IDLE_MINThe personal-notebook container: how many kernels it holds, its process ceiling, idle minutes before a kernel is stopped.60, 2048, 30
    KERNEL_ROOM_IDLE_MINMinutes an empty class (nobody inside, nothing computing) keeps its kernels; 0 keeps them. Also set on the Resources tab.120
    KERNEL_ENV, KERNEL_PRELOADThe default environment; which environments to prepare at start.base; same as KERNEL_ENV
    KERNEL_GPUSCards for GPU rooms and, of those, the ones given to competition submissions.all detected
    COMPETITION_SLOTSHow many submissions run at once: a number from 1 to 64, or auto, worked out from the machine. Also set on the Resources tab.auto
    COMPETITION_GPUS, COMPETITION_GPU_PER_DEVICEThe cards given to competition submissions (entries of KERNEL_GPUS, comma-separated) and how many submissions share one card at a time, 1 to 8; a change reaches the runs taken after it, and a submission already on a card keeps its share. Also set on the Resources tab; see a card for submissions.none; 1
    OPEN_SEMINAR_CREATIONAllow creating rooms through the API without a teacher's sign-in.false
    SESSION_SECRETThe link-signing key; a new value invalidates every link handed out.generated, data/session-secret
    Oracle
    AI_PROVIDERopenai, ollama, vllm, openrouter or custom.openai
    OPENAI_API_KEYThe model key; without it the Oracle is off (Ollama and vLLM work without a key).—
    OPENAI_BASE_URLThe model's address.https://api.openai.com/v1
    OPENAI_MODELThe model.gpt-4o-mini
    Sign-in through a proxy (SSO)
    AUTH_JWT_JWKS_URL, AUTH_JWT_AUDIENCE, AUTH_JWT_ISSUER and the other AUTH_JWT_*Sign-in through Teleport or another proxy that signs a JWT: teachers are recognised by email, students join under their own name. How it works and which token fields are read is in the Kubernetes guide; on a machine the same variables go to colloq-host up. When Teleport publishes the app from this same machine: COLLOQ_TUNNEL=none, and PUBLIC_URL is the app's address in Teleport.off
    
 
    Backups
    colloq-host backup                # a backup now
 colloq-host backup-timer on       # every day at 03:30; another time: COLLOQ_BACKUP_AT=02:00
 colloq-host backup-timer status
    The timer is a systemd unit: the backup runs in the machine's time zone, and one missed while the machine was off runs as soon as it is back. A backup is a pair of files in /workspace/colloq/backups/: a consistent snapshot of the database, colloq-<time>.db, and an archive, colloq-<time>-files.tar.gz, with room files, output images, class page files, competition data and submissions, package sets, the link-signing key, the setup token and the package lists of your own environments. The files are copied live, and during a class they can change while being copied, so schedule the timer for the night. The 14 most recent backups are kept (BACKUP_KEEP); older ones are deleted after a successful backup. Kernel images are not in the backup: they are pulled or built again.
    A backup is the key to the server: it holds the signing key, the setup token and the database with the teachers' sign-in keys and the AI key, if it was entered in the panel. The files are created with mode 0600. Copy them off the machine with your own tools (rsync, restic or borg into university storage) and keep them encrypted: the VM's disk is not a backup. Colloq can encrypt them itself: set BACKUP_AGE_RECIPIENT to an age public key, and both parts of a backup are written as .age. Keep the private key off the machine; to restore, name it: BACKUP_AGE_IDENTITY=/root/colloq-backup.key REPLACE=1 colloq-host restore ….
    To restore, put the pair into /workspace/colloq/backups/ and run the command below. colloq-host stops Colloq, restores the database and files and starts it again; REPLACE=1 is needed when the machine already has a database. Test a restore on a spare VM at least once a term.
    REPLACE=1 colloq-host restore backups/colloq-20260930-033000.db
    
diff --git a/docs/pages/en/troubleshooting.html b/docs/pages/en/troubleshooting.html
index 0a001f9e..8d60397a 100644
--- a/docs/pages/en/troubleshooting.html
+++ b/docs/pages/en/troubleshooting.html
@@ -6,7 +6,7 @@ 
    The notebook opens but Python doesn't run
    sudo sc
 sudo k3s kubectl -n colloq get events --sort-by=.lastTimestamp
    Check that the broker is running, the pinned image digest is available, there is enough CPU, memory or GPU, and the workspace mount is correct. ImagePullBackOff usually means checking the registry and credentials; Pending means looking for scheduling reasons in the events. A room's memory is reserved in full, so Insufficient memory in the events means the node lacks memory already promised to rooms: lower the memory of this room or of neighbouring rooms in “Resources” (see room resources). The broker waits 20 seconds for such a Pod, then deletes it and reports the cause in the room: the teacher sees how much memory, how many cores or which GPU couldn't be found, students see a short note without the details. In the app log this is a did not start … [unschedulable: memory] line (or cpu, gpu, other). The broker's health check tests API access, not the whole execution path.
    Don't swap an unavailable revision for another one without a deliberate decision, and don't connect rooms to a shared Jupyter. Once the cause is fixed, run sudo scripts/cluster.sh smoke and a real cell through the app.
    On a server running the ready-made image, start with colloq-host status and colloq-host logs; see Monitoring, logs and reboots.
    In a local class (pip install colloq), start with colloq doctor: it checks programs, Docker, the port, the kernel image, files and disk space. colloq logs shows the class log, and docker ps --filter name=colloq-room- lists room containers. If the room says it cannot block access to local addresses, Docker refused the short privileged container that installs this block; the message names the cause, and a class on your own computer explains the block and the COLLOQ_ROOM_NETWORK=open way out.
    
 
    NameError or changed data
    A setup cell may not have been run, the kernel may have restarted, or another participant may have changed shared values. Check the run order. In a Council, each attempt works on personal copies of the data; what carries over between attempts is files, module settings and anything that could not be copied — an attempt is told about the last of these in its own output.
    
 
    A package won't install or data won't download
    In production, kernel traffic out is limited to DNS by default, and the root filesystem is read-only. Prepare the library in an environment image, and upload the data to the room or use an outbound profile the operator has approved. Don't disable network protection for the whole namespace for the sake of one import.
    In a local class the internet is open to rooms, but local addresses aren't: a package mirror, a database or a data server on your own network is out of reach, and the connection is refused at once. Put the data into the room, or, if the class is trusted, add COLLOQ_ROOM_NETWORK=open to .env and restart (see a class on your own computer).
    
-
    CUDA is unavailable
    Check the guest VM's driver, the NVIDIA RuntimeClass, allocatable devices, whether a device is free and the catalog's GPU metadata. A CPU environment doesn't become a GPU environment on its own.
    nvidia-smi
+
    CUDA is unavailable
    Check the guest VM's driver, the NVIDIA RuntimeClass, allocatable devices, whether a device is free and the catalog's GPU metadata. A CPU environment doesn't become a GPU environment on its own. The shipped GPU environments carry torch 2.14.1 built for CUDA 13.0, which needs driver 580 or newer: nvidia-smi names the driver's version and its CUDA in its header, and on an older driver PyTorch imports but torch.cuda.is_available() is false.
    nvidia-smi
 sudo k3s kubectl get runtimeclass
 sudo k3s kubectl describe node
 sudo scripts/cluster.sh gpu-preflight
    If a room is using the GPU, the preflight doesn't evict it. Repeat the check at an agreed time when the card is free. A real CUDA run is necessary even when nvidia-smi looks right.
    
diff --git a/docs/pages/en/vast.html b/docs/pages/en/vast.html
index 2601e4c9..c4082786 100644
--- a/docs/pages/en/vast.html
+++ b/docs/pages/en/vast.html
@@ -1,4 +1,4 @@
-
    The ready-made image: make vast-vm
    The quick way is make vast-vm HOST=class1 from a checkout of the repository, with VAST_TOKEN and an SSH key in .env. It rents a Vast VM over the API, starts the ready-made colloq-vast image on it (the same image as colloq-server) under colloq-host, and prints the class address and the owner link. Each room gets its own kernel container on that VM's Docker. The class address usually comes from your relay; fill in the settings — the relay token, the AI key, COLLOQ_IMAGE — once in backups/vast-onstart.sh. On the VM itself, looking after it — colloq-host status, backup, update — works as on a university server; the rental details are in deploy/vast/README.md. The data lives on the VM's disk and goes with the rental: copy the backups off before you delete it.
    
+
    The ready-made image: make vast-vm
    The quick way is make vast-vm HOST=class1 from a checkout of the repository, with VAST_TOKEN and an SSH key in .env. It rents a Vast VM over the API, starts the ready-made colloq-vast image on it (the same image as colloq-server) under colloq-host, and prints the class address and the owner link. Each room gets its own kernel container on that VM's Docker. The class address usually comes from your relay; fill in the settings — the relay token, the AI key, COLLOQ_IMAGE — once in backups/vast-onstart.sh. On the VM itself, looking after it — colloq-host status, backup, update — works as on a university server; the rental details are in deploy/vast/README.md. The data lives on the VM's disk and goes with the rental: copy the backups off before you delete it.
    For GPU rooms or GPU submissions, rent the cards and the disk the CUDA images need: make vast-vm HOST=class1 GPUS=2 GPU_RAM=24 DISK=100 asks for at least two cards of 24 GB each and 100 GB of disk, since base-gpu (about 9.9 GB) and kaggle-gpu (about 2.8 GB more) are built on the VM itself. Whatever the class, make vast-vm rents only offers whose Max CUDA is 13.0 or more: the GPU environments' torch 2.14.1 is built for CUDA 13.0 and needs an NVIDIA driver of 580 or newer. DISK= and MAX_PRICE= override .env for that one rental, and OFFER= rents the offer with that number wherever it stands in the market. In backups/vast-onstart.sh set KERNEL_PRELOAD=base,base-gpu,kaggle-gpu, and once the class is up give submissions their cards in the panel; see a card for submissions.
    
 
    Before you rent
    The k3s path uses a full Vast VM with systemd, not a regular Docker instance. The VM runs a single k3s node. Check that VM offers are available, and check for a working NVIDIA driver, a registered SSH key, the price and disk space.
    On a trusted operator machine you need the release manifest and a clone of the public repository github.com/colloq-edu/colloq that contains its exact sourceCommit. vast-up packs the tools from exactly that commit. The app never gets your Vast account key.
    In the configuration, prepare VAST_TOKEN, an SSH key and the settings of an existing relay. For private images, set VAST_REGISTRY_CONFIG with separate pull-only credentials and VAST_REGISTRY_READ_ONLY=1. The file must contain only Docker auths, with no credential helpers. A syntax check doesn't prove the credentials work — the preflight checks the pull.
    
 
    Launch and monitor
    For a release whose images really are available anonymously:
    VAST_PUBLIC_IMAGES=1 RELEASE=/absolute/path/release.json \
   NAME=hse HOST=hse.example.edu make vast-up
diff --git a/docs/pages/ru/competitions.html b/docs/pages/ru/competitions.html
index f7356065..ba2a69e4 100644
--- a/docs/pages/ru/competitions.html
+++ b/docs/pages/ru/competitions.html
@@ -24,12 +24,14 @@ 
    Публичная и приватная части
    Мет
 
 
    Скрытый тест
    Открытый data/test.csv класс скачивает, и по нему легко подогнать ответ: посчитать строки, выписать id. Скрытый тест — файл, который участник не скачивает: на каждой проверке тетрадь читает его по тому же пути, вместо открытого примера. Модель проверяется на строках, которых никто не видел.
    1. В редакторе, в разделе «Скрытый тест», загрузите файл своей кнопкой — не через открытые данные. В поле «Путь при проверке» укажите имя в data/: по умолчанию это имя самого файла. Файл с именем открытого примера подменяет пример, файл с новым именем добавляется к данным.
    2. Пример остаётся у участников: его скачивают и на нём отлаживают тетрадь. Сверьте колонки — редактор показывает, совпадают ли они с примером. Тетрадь, написанная под пример, упадёт на тесте с другими колонками.
    3. Ответы в solution.csv должны быть к строкам скрытого теста.
    4. Проверьте сэмпл-тетрадь заново: смена скрытого теста делает её прошлую проверку устаревшей.
    Каждый прогон получает собственный каталог data/: открытые файлы и скрытые на их месте. data/test.csv и /data/test.csv читают скрытый тест, метрика его не видит вовсе. Файл, заменённый посреди прогона, достаётся только следующим прогонам. Потолок — 200 МБ и 10 файлов, отдельно от открытых данных.
    Участник видит имя скрытого файла и число строк: в списке данных у примера стоит отметка «ПРИМЕР», а в условиях проверки — строка о скрытом тесте. Содержимого не отдаёт ни одна ссылка, ни участнику, ни вам: скачать скрытый тест нельзя.
    Пока тетрадь работает, скрытые строки у неё в памяти, и любой вывод может их вынести: test.head(), текст KeyError со значением, сообщение метрики, которое цитирует ответ. Поэтому в разделе «Что показывать участнику при проверке» два варианта:
    • «Только статус и тип ошибки» — по умолчанию. Участник видит исход, номер упавшей ячейки и тип исключения из короткого списка обычных (KeyError, ValueError…). Ни вывода ячеек, ни текста ошибки, ни трассировки; ссылка на тетрадь отдаёт тетрадь в том виде, в каком её прислали, а не исполненную. Отказ метрики пересказывается нашими словами, без значений, id и чисел из ответа.
    • «Полный вывод» — как без скрытого теста. Включайте, только если раскрыть эти строки не страшно: подогнать ответ по ним станет можно.
    Вы в ленте всегда видите весь вывод и исполненную тетрадь. Полный текст хранится при любом варианте, поэтому переключение действует сразу на все посылки, без повторного исполнения.
    
 
-
    Как исполняется посылка
    Участник нажимает «ВЫБРАТЬ ФАЙЛ» и отправляет тетрадь. Загрузки прямо из тетради занятия нет: только файл.
    Открытые файлы участник скачивает по одному или кнопкой «Скачать всё» над списком: в архиве папка data/ со всеми файлами и стартовые тетради рядом с ней, как при проверке, поэтому pd.read_csv('data/train.csv') работает сразу после распаковки. Скрытых ответов и скрытого теста в архиве нет.
    1. Посылка встаёт в очередь инстанса.
    2. В одноразовом контейнере без сети тетрадь исполняется целиком, ячейка за ячейкой, с нуля: сохранённых переменных нет, каталог data/ примонтирован только на чтение. Тетрадь обязана записать submission.csv.
    3. Файл забирается из контейнера, контейнер сносится.
    4. Во втором одноразовом контейнере — уже без данных участника и без его кода — запускается ваша метрика. Скрытые ответы бывают только здесь.
    Пределы соревнования (время на тетрадь, память, ядра) стоят в редакторе. По времени контейнер убивают снаружи, по памяти его ограничивает docker. Участник видит движение этапов вживую и, если тетрадь упала, свою трассировку целиком. У посылки есть ссылка «Скачать тетрадь с выводом»: это тетрадь в том виде, в каком её исполнил сервер, — у упавшей тоже, до ячейки, на которой она остановилась. Вывод в ней только этого прогона: присланный с ноутбука стирается перед запуском.
    ИсходЧто видит участник
    ГОТОВОПубличный результат и своё место.
    УПАЛА ТЕТРАДЬ«Упала на ячейке 5 из 14» и свою трассировку дословно.
    ЛИМИТ ВРЕМЕНИ«Не уложилась в 10 минут, ячейка 6 из 50».
    НЕ ХВАТИЛО ПАМЯТИ«Заняла больше 4 ГБ, ячейка 2».
    ОТВЕТ НЕ ПРИНЯТ«Дошла до конца, но submission.csv не оставила», «В ответе нет 3 строк, например id=17» — или текст вашего ParticipantVisibleError.
    УПАЛА МЕТРИКА«Проверяющий код упал» без подробностей; трассировку видите вы, и посылка ждёт пересчёта.
    Так выглядит соревнование без скрытого теста. Со скрытым тестом участник по умолчанию видит упавшую посылку коротко: исход, номер ячейки и тип исключения, без вывода и трассировки, см. скрытый тест.
    
+
    Как исполняется посылка
    Участник нажимает «ВЫБРАТЬ ФАЙЛ» и отправляет тетрадь. Загрузки прямо из тетради занятия нет: только файл.
    Открытые файлы участник скачивает по одному или кнопкой «Скачать всё» над списком: в архиве папка data/ со всеми файлами и стартовые тетради рядом с ней, как при проверке, поэтому pd.read_csv('data/train.csv') работает сразу после распаковки. Скрытых ответов и скрытого теста в архиве нет.
    1. Посылка встаёт в очередь инстанса.
    2. В одноразовом контейнере без сети тетрадь исполняется целиком, ячейка за ячейкой, с нуля: сохранённых переменных нет, каталог data/ примонтирован только на чтение. Тетрадь обязана записать submission.csv.
    3. Файл забирается из контейнера, контейнер сносится.
    4. Во втором одноразовом контейнере — уже без данных участника и без его кода — запускается ваша метрика. Скрытые ответы бывают только здесь.
    Пределы соревнования (время на тетрадь, память, ядра и переключатель «Видеокарта») стоят в редакторе. По времени контейнер убивают снаружи, по памяти его ограничивает docker. Участник видит движение этапов вживую и, если тетрадь упала, свою трассировку целиком. У посылки есть ссылка «Скачать тетрадь с выводом»: это тетрадь в том виде, в каком её исполнил сервер, — у упавшей тоже, до ячейки, на которой она остановилась. Вывод в ней только этого прогона: присланный с ноутбука стирается перед запуском.
    ИсходЧто видит участник
    ГОТОВОПубличный результат и своё место.
    УПАЛА ТЕТРАДЬ«Упала на ячейке 5 из 14» и свою трассировку дословно.
    ЛИМИТ ВРЕМЕНИ«Не уложилась в 10 минут, ячейка 6 из 50».
    НЕ ХВАТИЛО ПАМЯТИ«Заняла больше 4 ГБ, ячейка 2».
    ОТВЕТ НЕ ПРИНЯТ«Дошла до конца, но submission.csv не оставила», «В ответе нет 3 строк, например id=17» — или текст вашего ParticipantVisibleError.
    УПАЛА МЕТРИКА«Проверяющий код упал» без подробностей; трассировку видите вы, и посылка ждёт пересчёта.
    Так выглядит соревнование без скрытого теста. Со скрытым тестом участник по умолчанию видит упавшую посылку коротко: исход, номер ячейки и тип исключения, без вывода и трассировки, см. скрытый тест.
    
 
-
    Очередь и пульт
    Очередь одна на весь инстанс: она лежит в базе и переживает перезапуск сервера. Сколько посылок исполняется одновременно, по умолчанию считается по машине («Авто»): её ядра и память за вычетом 2 ядер и 2 ГБ, оставленных занятию, делятся на ядра и память посылки по умолчанию. Своё число владелец задаёт во вкладке «Ресурсы» панели, см. ресурсы. Очередь честная по людям: вторая посылка одного человека встаёт за первыми посылками остальных, поэтому один участник не займёт исполнитель на всю пару.
    Пульт соревнования (вкладка «Посылки») показывает, что исполняется сейчас, кто ждёт, пять чисел сводки и ленту посылок. В меню строки: открыть исполненную тетрадь, посмотреть весь вывод, исполнить заново, пересчитать метрику, не засчитывать. Полоса исполнителя в списке соревнований позволяет приостановить очередь, а у идущего прогона есть кнопка «Убить».
    Чтобы никто не подгонял решение под публичный лидерборд десятками посылок, поставьте дневную норму посылок. Норму тратят только посылки, получившие оценку. Упавшие — ошибка в тетради, лимит времени или памяти, непринятый ответ, сбой метрики или самого сервера — и отменённые её не тратят; посылка в очереди временно занимает место и возвращает его, если падает. От бесконечных падений защищает отдельный потолок упавших посылок: не больше 10 в день на участника или трёх норм, если это больше (сбои метрики в него не идут). Упёршийся в потолок участник видит свой отказ, отдельный от нормы. Без нормы (0) нет и потолка.
    
+
    Видеокарта для посылок
    Тетради, которая учит нейросеть, нужна видеокарта. Посылка получает долю карты, когда сошлись два условия: владелец сервера отдал посылкам карты, и соревнование карту просит.
    1. Владелец отдаёт карты. Во вкладке «Ресурсы» панели блок «Видеокарты для посылок» перечисляет карты машины (KERNEL_GPUS) с названиями и памятью. Отметьте те, что служат посылкам, и задайте, сколько посылок делят одну карту одновременно, от 1 до 8. Отданная посылкам карта уходит из пула комнат, поэтому занятие и посылка никогда не делят одну карту; занятие, которое держит карту в этот момент, сохраняет её, пока работает его контейнер, а посылки до тех пор её ждут; и наоборот, карта, возвращённая комнатам, начинает служить занятию, когда закончатся идущие на ней посылки. На машине, настроенной без панели, то же задают COMPETITION_GPUS (элементы KERNEL_GPUS через запятую) и COMPETITION_GPU_PER_DEVICE; сохранённое в панели главнее.
    2. Соревнование просит карту. В редакторе выберите окружение с CUDA (base-gpu, kaggle-gpu или своё поверх них) и включите переключатель «Видеокарта». Он есть только у таких окружений, и пока он включён, окружение нельзя сменить на окружение без CUDA. Карту получает только тетрадь. Метрика не получает никогда: у неё скрытые ответы, и учить ей нечего.
    Доля карты. Посылки на одной карте получают поровну её памяти: на карте 24 ГБ при трёх одновременно — по 8 ГБ. Если карты разного размера, обещанная доля — по наименьшей из них: её хватит на любой карте. Число посылок на карту можно менять и посреди занятия: идущая посылка сохраняет долю, с которой началась, а новую карта берёт, только когда её доля помещается рядом с их долями, — поэтому карта, которую одна посылка занимает целиком, после перехода с 1 на 2 никого больше не возьмёт, пока та не закончится.
    Как держат долю. С двух сторон. Внутри посылки PyTorch ограничен долей за вычетом 512 МБ на контекст CUDA, даже если какая-нибудь библиотека, например polars, ищет его раньше, чем тетрадь его импортирует: модель, которая выросла из ограничения, падает в своей ячейке с torch.OutOfMemoryError, как на карте поменьше. CatBoost на GPU по умолчанию сразу берёт 95 % свободной памяти карты; в доле его gpu_ram_part — та часть карты, которую составляет доля за вычетом контекста CUDA (около 0,48 при двух посылках на карте 24 ГБ), если тетрадь не задала свою. XGBoost на device="cuda" берёт память по ходу дела, а не заранее. TensorFlow и JAX в своём окружении не ограничиваются: по умолчанию оба сразу занимают большую часть карты, и исполнитель останавливает их за это, если тетрадь не придержит их до импорта (TF_FORCE_GPU_ALLOW_GROWTH=true, XLA_PYTHON_CLIENT_PREALLOCATE=false). Снаружи исполнитель каждые 3 секунды смотрит, сколько памяти карты на самом деле держит контейнер посылки, PyTorch это или нет, и останавливает посылку, которая дважды подряд превысила долю больше чем на 256 МБ. Смотрит он от root, которому код тетради не может послать сигнал. А посылку, карту которой 4 раза подряд не удалось прочитать (около 12 секунд), тоже останавливают: прогон, за которым никто не смотрит, может держать всю карту. Если на карте одна посылка, карта целиком её, и ничего не ограничивается и не проверяется. Если карта не сообщила свой объём, обещать нечего: панель пишет, что размер неизвестен, и ничего не ограничивается и не проверяется тоже.
    Что видит участник. На странице соревнования доля карты стоит рядом с остальными пределами. Тетрадь, упёршаяся в ограничение изнутри, — это «Упала тетрадь» с тем самым torch.OutOfMemoryError и нашей строкой о её доле; посылка, остановленная снаружи, — «Не хватило памяти» с её долей и тем, сколько она заняла. Со скрытым тестом при варианте «Только статус и тип ошибки», который стоит по умолчанию, в сообщении только доля: сколько заняла посылка — число, которое выбрала её тетрадь (см. скрытый тест). Посылка, карта которой перестала отвечать, останавливается как «Упала тетрадь» и считается упавшей: так бывает, когда тетрадь занимает процессор сотнями потоков или процессов или когда сама карта перестаёт отвечать. Если же карта не ответила за весь прогон ни разу, виноват хост, а не тетрадь: посылка заканчивается сбоем самого сервера, который ни во что не засчитывается, а в журнале преподавателя сказано проверить nvidia-smi внутри контейнера. Всплеск мимо ограничения может за секунды до второго замера отнять память у соседей по карте: посылку, которой не хватило видеопамяти (torch.OutOfMemoryError), пока исполнитель останавливал другую посылку на той же карте за выход за долю, один раз возвращают в очередь, с нашей строкой об этом, и эта попытка не засчитывается. Если посылкам не отдано ни одной карты, соревнование с «Видеокартой» сразу отказывает в загрузке и в проверке бейслайна и называет причину: никто не ждёт в очереди карту, которая не придёт.
    Чего доля не даёт. Посылки на одной карте пользуются её вычислениями по очереди: модель, которая держит карту занятой, при трёх посылках на карте учится примерно втрое медленнее, чем одна, а предел времени тетради считает это время. Доля бережёт память, а не скорость: поднимите предел времени или посадите на карту меньше посылок. Небольшая модель между своими шагами оставляет карту свободной и соседей почти не замечает, см. замер ниже. В очереди GPU-посылке нужны свободный исполнитель и свободная доля карты; пока она ждёт карту, процессорные посылки за ней проходят вперёд.
    Что карта даёт занятию. Замер 13 октября 2026 года на Vast VM с 61 vCPU, 98 ГБ памяти и 2 × RTX 4090 (по 24 ГБ): 25 участников в один момент отправили эталонное решение соревнования по распознаванию букв — небольшую свёрточную сеть; 2 ядра, 4 ГБ и 600 секунд на посылку; kaggle-gpu, — и каждая посылка получила оценку. Прогон — время, которое посылка исполнялась; вердикт считается от её загрузки.
    Карты для посылокПрогон, медианаВердикт, медианаВердикт, последний
    нет: только процессор, на 0.21.0 (22 исполнителя по 2 ядра)159 с162 с235 с
    1 карта × 425 с102 с175 с
    2 карты × 429 с58 с88 с
    1 карта × 8 (доля около 3 ГБ)29 с60 с117 с
    2 карты × 831 с36 с62 с
    Карта сокращает один прогон этой тетради в пять-шесть раз, и дальше занятие ждёт очередь, а не прогон: восемь посылок на карте шли почти так же быстро, как четыре, а две карты по восемь выдали вердикт всем 25 примерно за минуту. Тяжёлой модели лишние посылки на карте помогают меньше, и ей ещё нужно уместиться в долю поменьше. При смешанной нагрузке — 10 GPU-посылок и 4 процессорных на той же машине — очередь держала своё правило: процессорные посылки не ждали за GPU-посылками.
    На Kubernetes долю определяет кластер: значение чарта gpu.competitions.resource называет ресурс device plugin, который просит GPU-посылка, — целую карту, реплику time-slicing или часть MIG; см. эксплуатацию. Сколько GPU-посылок одновременно уходит в кластер, задаёт gpu.competitions.slots, по одной, пока оно не задано: очередь держит там столько и пропускает процессорные посылки мимо остальных, как мимо занятой карты здесь. Сам Colloq там ничего не ограничивает и не проверяет: у части MIG своя память, а реплики time-slicing делят память карты без всякой границы между собой. И карта там не уходит из пула комнат: GPU-комнаты просят nvidia.com/gpu, поэтому с этим именем посылки берут единицы с карт комнат, и GPU-комната, которой не хватило свободной карты, не запустится, пока их держат посылки. Если на занятиях работают GPU-комнаты, дайте посылкам собственный ресурс, см. узлы с GPU.
    
+
+
    Очередь и пульт
    Очередь одна на весь инстанс: она лежит в базе и переживает перезапуск сервера. Сколько посылок исполняется одновременно, по умолчанию считается по машине («Авто»): её ядра и память за вычетом 2 ядер и 2 ГБ, оставленных занятию, делятся на ядра и память посылки по умолчанию. Своё число владелец задаёт во вкладке «Ресурсы» панели, см. ресурсы. Очередь честная по людям: вторая посылка одного человека встаёт за первыми посылками остальных, поэтому один участник не займёт исполнитель на всю пару.
    Пульт соревнования (вкладка «Посылки») показывает, что исполняется сейчас, кто ждёт, пять чисел сводки и ленту посылок. В меню строки: открыть исполненную тетрадь, посмотреть весь вывод, исполнить заново, пересчитать метрику, не засчитывать. Полоса исполнителя в списке соревнований позволяет приостановить очередь, а у идущего прогона есть кнопка «Убить». Владелец удаляет соревнование пунктом «Удалить соревнование» в меню его строки в списке, и сначала уходят его прогоны: ждущие посылки покидают очередь, а идущие убиваются, и удаление ждёт их остановки, до 30 секунд на каждую. Если какая-то за это время не остановилась, ничего не удаляется, хотя ждавшие посылки остаются отменёнными, и удаление можно повторить через минуту.
    Чтобы никто не подгонял решение под публичный лидерборд десятками посылок, поставьте дневную норму посылок. Норму тратят только посылки, получившие оценку. Упавшие — ошибка в тетради, лимит времени или памяти, непринятый ответ, сбой метрики или самого сервера — и отменённые её не тратят; посылка в очереди временно занимает место и возвращает его, если падает. От бесконечных падений защищает отдельный потолок упавших посылок: не больше 10 в день на участника или трёх норм, если это больше (сбои метрики в него не идут). Упёршийся в потолок участник видит свой отказ, отдельный от нормы. Без нормы (0) нет и потолка.
    
 
 
    Участники и ключ входа
    Участник соревнования — не житель комнаты: его личность принадлежит всему инстансу. Вступая, человек указывает логин Telegram или почту — под этим именем он стоит в лидерборде — и получает ключ входа вида K7Q-M2X-9FD и ссылку к нему. Тем же ключом он возвращается к своим посылкам с другого устройства, из другого браузера или после очистки данных. Логин или почта — это один человек: под именем, которое в соревновании уже есть, вступить нельзя. Отказ предлагает войти по своему ключу, и второй личности с другим ключом не появляется.
    Ключ лежит в карточке «ВАШ КЛЮЧ ВХОДА» на странице соревнований — её видно каждый раз, а не однажды. В базе самого ключа нет: хранятся его отпечаток и копия, зашифрованная секретом инстанса, поэтому украденная база без секрета не открывает ни одного входа.
    Если участник ключ потерял, выдайте новый на вкладке «Участники». Старый перестаёт действовать в ту же секунду — вместе со всеми вкладками, которые были на нём открыты.
    
 
 
    Дедлайн и разбор
    Когда дедлайн проходит, приём посылок закрывается сам, а итоговый лидерборд открывается — если в настройках выбрано «открыть сам после дедлайна». Второй вариант — «Открою вручную — на разборе»: тогда итоги показывает кнопка, и вы решаете, в какую минуту пары их увидят.
    «Завершить сейчас» закрывает приём раньше срока. После дедлайна нажимать её не нужно: соревнование и так считается завершённым — в списке у участников и в панели оно стоит среди завершённых, а в шапке пульта вместо «Завершить сейчас» появляются «Закрыть поздний приём» или «Принимать поздние» и, если итоги открываются вручную, «Открыть приватный лидерборд».
    Тем, кто доделывает решение после звонка, поможет переключатель «Поздние посылки» в разделе «Сроки и зачёт». С ним после дедлайна и после «Завершить сейчас» посылки принимаются дальше: исполняются, получают публичный счёт и тратят дневной лимит по тем же правилам, что и обычные. Но в зачёт поздняя посылка не идёт никогда: её нет в местах, лидербордах, сводке и ожидании итогов, выбрать её для зачёта нельзя. Итоговый счёт участник увидит, когда откроются итоги. У него на странице она отмечена «после дедлайна · вне зачёта», а на пульте у неё отметка «ПОЗДНЯЯ» и свой фильтр «Поздние». Поздней становится посылка, загрузка которой началась после дедлайна; начатая вовремя и докачанная в пределах двух минут остаётся в зачёте. Ждущую проверки посылку, отправленную вовремя, поздняя не заменяет. Пока приём поздних открыт, это видно в списках: у участников соревнование стоит среди завершённых с отметкой «приём поздних посылок открыт» и кнопкой, а в панели под «ЗАВЕРШЕНО» написано «поздние принимаются» и сколько их пришло.
    Если на идущем соревновании перенести дедлайн позже, посылки, отправленные до нового дедлайна, становятся обычными: попадают в лидерборд и в зачёт. Перенос раньше никого поздним не делает. После «Завершить сейчас» перенос ничего не меняет: итоги закрыты вручную.
    На разборе пригодится «Лидерборд на проектор» — отдельное окно с одной таблицей, без имён файлов и служебных отметок.
    
 
-
    Дополнительные пакеты
    Во вкладке «Пакеты» участник видит закреплённую версию Python и полный состав базы. Дополнительные библиотеки можно указать по одной на строку или загрузить из requirements.txt. Комментарии — отдельной строкой или после запроса (numpy==1.26  # для метрики) — и пустые строки не входят в лимит 32 запросов; весь файл ограничен 8 КиБ. Сервер заранее проверяет совместимость, скачивает готовые wheels из PyPI и проверяет установку без сети. Версии базовых пакетов менять нельзя.
    Готовый набор выбирается перед явным подтверждением отправки тетради. Он и базовый образ закрепляются за посылкой; изменение выбора не меняет повтор старой посылки. Набор ставится в память посылки при каждом запуске и считается в её лимит: ему можно не больше половины, остальное — ядру, данным и рабочей папке. Набор больше этого не выбрать, и участник видит, сколько он займёт. На Docker-пути для подготовки требуется Docker Engine 28 или новее. Во время исполнения тетради и метрики интернет выключен.
    Проверки и повторное исполнение
    Каждая попытка получает отдельные каталоги и идентификатор. Пересчёт активной посылки ждёт окончания текущей попытки; старый CSV не может стать результатом неудачного нового запуска. Изменение метрики, данных, базового окружения или стартовой тетради требует актуальной проверки бейслайна. После изменения данных или окружения одного пересчёта метрики недостаточно: тетрадь нужно исполнить снова.
    Доступность исполнителя
    Посылки и подготовка пакетов исполняются там же, где ядра комнат: на Docker-пути (готовый образ сервера, colloq start) — в одноразовых контейнерах того же Docker, на k3s — в подах, которые запускает брокер. Если исполнитель или подготовка пакетов недоступны, интерфейс называет причину и не принимает работу в очередь; /api/health сообщает о каждой возможности отдельно.
    
+
    Дополнительные пакеты
    Во вкладке «Пакеты» участник видит закреплённую версию Python и полный состав базы. Дополнительные библиотеки можно указать по одной на строку или загрузить из requirements.txt. Комментарии — отдельной строкой или после запроса (numpy==1.26  # для метрики) — и пустые строки не входят в лимит 32 запросов; весь файл ограничен 8 КиБ. Сервер заранее проверяет совместимость, скачивает готовые wheels из PyPI и проверяет установку без сети. Версии базовых пакетов менять нельзя.
    Готовый набор выбирается перед явным подтверждением отправки тетради. Он и базовый образ закрепляются за посылкой; изменение выбора не меняет повтор старой посылки. Набор ставится в память посылки при каждом запуске и считается в её лимит: ему можно не больше половины, остальное — ядру, данным и рабочей папке. Набор больше этого не выбрать, и участник видит, сколько он займёт. На Docker-пути для подготовки требуется Docker Engine 28 или новее. Во время исполнения тетради и метрики интернет выключен.
    Проверки и повторное исполнение
    Каждая попытка получает отдельные каталоги и идентификатор. Пересчёт активной посылки ждёт окончания текущей попытки; старый CSV не может стать результатом неудачного нового запуска. Изменение метрики, данных, базового окружения или стартовой тетради требует актуальной проверки бейслайна. После изменения данных или окружения одного пересчёта метрики недостаточно: тетрадь нужно исполнить снова.
    Доступность исполнителя
    Посылки и подготовка пакетов исполняются там же, где ядра комнат: на Docker-пути (готовый образ сервера, colloq start) — в одноразовых контейнерах того же Docker, на k3s — в подах, которые запускает брокер. Если исполнитель или подготовка пакетов недоступны, интерфейс называет причину и не принимает работу в очередь; /api/health сообщает о каждой возможности отдельно. Соревнованию с «Видеокартой» нужна ещё и доля карты: если посылкам не отдано ни одной карты (на Kubernetes — не задан GPU-ресурс для посылок), оно отказывает в загрузке и в проверке бейслайна с этой причиной.
    
diff --git a/docs/pages/ru/environments.html b/docs/pages/ru/environments.html
index 6dc7313c..11633591 100644
--- a/docs/pages/ru/environments.html
+++ b/docs/pages/ru/environments.html
@@ -5,7 +5,7 @@ 
    Сборка выпуска сопровождающим<
   --k3s-version EXACT_TESTED_K3S_VERSION --environments base,kaggle-base
    Для GPU дополнительно укажите --gpu-toolkit-version EXACT_NVIDIA_APT_VERSION и --gpu-device-plugin-image NVIDIA_PLUGIN_IMAGE_AT_SHA256. Авторизуйтесь в registry на машине сборки. Версия k3s нужна v1.33 или новее: память комнаты меняется во время занятия через подресурс pods/resize, и проверка манифеста отвергает более старую. Публикация артефактов сама по себе не является проверкой на целевой VM.
    
 
    Память и процессор занятия
    Раздел «Ресурсы» есть в форме нового занятия и в настройках существующего (меню занятия → «Настройки»). Поле «Память ядра комнаты, ГБ» меняется с шагом 0,5 ГБ: от 0,5 ГБ до памяти машины за вычетом 1 ГБ. Поле «Процессор, ядер» принимает целое число от 1 до числа ядер машины. Чтобы вернуть общее значение, очистите поле или нажмите «вернуть умолчание окружения» либо «вернуть умолчание инстанса».
    Подсказка под полем показывает объём памяти, свободную память и умолчание выбранного окружения. Если Docker работает в собственной виртуальной машине, как Docker Desktop или Colima на macOS, подсказка начинается словами «Комнаты живут в Docker». Тогда объём, число ядер и верхнюю границу полей задаёт Docker, а не компьютер. Свободной считается память Docker за вычетом 1 ГБ и лимитов уже работающих комнат. Если свободную память узнать нельзя, подсказка её не показывает. Лимит больше свободной памяти форма не запрещает, но предупреждает, что ядро может не подняться. У работающей комнаты со свободной памятью сравнивается только прибавка к тому, что у неё уже есть. Если на машине есть видеокарта NVIDIA, раздел показывает её и то, использует ли карту выбранное окружение. Видеопамять не ограничивается: её делят все комнаты на карте.
    В Docker новый лимит памяти применяется к работающему ядру сразу, без перезапуска; возврат к умолчанию окружения вступит в силу при следующем запуске контейнера комнаты. Новое число ядер тоже применяется к работающему ядру сразу, без перезапуска, — и возврат к умолчанию инстанса тоже. Число потоков numpy и torch (OMP_NUM_THREADS и соседние переменные) задаётся при создании контейнера комнаты и живому контейнеру уже не меняется, даже перезапуском ядра: перезапущенное ядро наследует окружение того же контейнера. Новое число потоков получит следующий контейнер комнаты — например, когда комната простоит пустой два часа и её откроют снова. До тех пор число потоков можно поднять прямо в тетради: threadpoolctl.threadpool_limits(n) для numpy и scikit-learn, torch.set_num_threads(n) для torch. Без KERNEL_MEM умолчание — 4 ГБ, для GPU-окружения — 16 ГБ. KERNEL_MEM задаёт умолчание для обычных окружений (GPU-окружение остаётся на 16 ГБ), KERNEL_MEM_<ОКРУЖЕНИЕ> — для одного (например, KERNEL_MEM_BASE_GPU=16g), KERNEL_CPUS — число ядер (по умолчанию 2). Файл .env, который создаёт colloq start, уже содержит KERNEL_MEM=4g. Владелец может поменять эти умолчания сервера без перезапуска (/api/admin/resources): сохранённое значение главнее .env, пока его не сбросят. Те же числа памяти и ядер получает второй контейнер занятия — тот, где считаются личные тетради студентов; cgroup у него свой, так что жадный черновик не отнимает память у лекции. GPU ему не выдаётся никогда. Сколько ядер может жить в нём одновременно, задаёт KERNEL_OWN_MAX (по умолчанию 60): сверх этого запуск в личной тетради отказывает словами. Потолок процессов у него свой, KERNEL_OWN_PIDS (по умолчанию 2048), — ядер в нём десятки, и комнатных 512 не хватило бы уже на тридцати. Ядро личной тетради, которое ничего не считало KERNEL_OWN_IDLE_MIN минут (по умолчанию 30; 0 — не гасить), останавливается: тетрадь читает об этом в журнале ядра, следующий запуск поднимает его заново, а контейнер уходит с последним ядром в нём.
    Запущенный контейнер занятия держит весь свой лимит памяти, даже если внутри почти пусто: пока он работает, другим занятиям эту память не отдать. Занятие, в котором никого нет и ничего не считается, отпускает ядра через KERNEL_ROOM_IDLE_MIN минут (по умолчанию 120; 0 — держать) — срок меняется и во вкладке «Ресурсы» → «Комната по умолчанию». Там же, в разделе «Работают сейчас», видно, какие занятия сейчас держат память, сколько из неё реально занято и кто внутри; владелец сервера может остановить ядро любого занятия или все простаивающие разом. Остановленное ядро теряет переменные, файлы остаются, а следующий запуск ячейки поднимает его снова. Как это устроено в k3s-установке, описано ниже.
    
 
    Ресурсы комнаты
    В k3s-установке Pod комнаты по умолчанию получает 2Gi памяти, 2 CPU и 2Gi ephemeral storage; requests равны limits. Память и число CPU из раздела «Ресурсы» занятия доходят до Pod, и их можно поднять или опустить во время занятия: брокер меняет их работающему Pod на месте, ядро не перезапускается, переменные остаются. Кнопки «вернуть умолчание окружения» и «вернуть умолчание инстанса» тоже действуют сразу. Число потоков numpy и torch Pod получает от своего лимита CPU при запуске и, как в Docker, живому ядру его уже не меняет — ни после изменения на месте, ни после перезапуска ядра; новое число получит следующий Pod комнаты. Умолчание брокера одно для всех окружений, включая GPU, — его и показывает подсказка формы. Верхняя граница поля — память узла за вычетом 1 ГБ. Умолчание и потолок задают переменные брокера RUNTIME_KERNEL_MEMORY и RUNTIME_KERNEL_MEMORY_MAX: их пишут в тот же instance.env, что и настройки приложения (см. конфигурацию установки), например RUNTIME_KERNEL_MEMORY=4Gi. Значения — целые Mi или Gi от 64Mi до 256Gi, потолок не ниже умолчания; установщик проверяет их до остановки комнат и сохраняет при обновлениях. Изменение на месте требует Kubernetes 1.33 или новее, и выпуск с более старым k3s не проходит проверку. Это пределы ресурсов, а не обещание определённого числа студентов или скорости обучения модели.
    Память и ядра комнаты резервируются целиком, даже если Python их не трогает. Поэтому свободной подсказка считает память узла за вычетом 1 ГБ и памяти, уже отданной работающим комнатам. Если узлу сейчас нечем покрыть прибавку памяти или ядер, новое значение сохраняется и ждёт: Kubernetes применит его сам, когда ресурс освободится, например когда остановится соседняя комната. Больше, чем узел может дать вообще, брокер не выставляет: работающий Pod остаётся с прежним значением. Новой комнате ждать негде: если узлу нечем покрыть её память, процессор или видеокарту, через 20 секунд подъём останавливается. Преподаватель видит в комнате, сколько она просила и что уменьшить или закрыть; студенты — короткое сообщение, что Python сейчас не запускается и преподаватель видит причину. Если коду не хватило памяти, в k3s завершается весь Pod комнаты, а не только ядро, как в Docker; следующий запуск кода поднимает новый Pod, уже с новым значением памяти.
    Память /dev/shm входит в лимит Pod: по умолчанию 64 MiB у CPU-окружения и 1 GiB у GPU. Установщик задаёт лимит 256 PID на Pod. Namespace ResourceQuota и дополнительные правила допуска нагрузки оператор настраивает отдельно; готовой ResourceQuota в комплекте нет. Для workspace нет принудительной покомнатной файловой квоты; следите за отдельным файловым разделом и свободным местом.
    
-
    GPU-окружения
    Каждая GPU-комната запрашивает один nvidia.com/gpu и RuntimeClass nvidia. Совместное использование устройства не включается автоматически. Несколько одновременных GPU-комнат требуют соответствующего числа доступных устройств либо отдельно спроектированной и проверенной политики sharing.
    Установщик сохраняет драйвер хоста, ставит закреплённые NVIDIA Container Toolkit и device plugin и проверяет реальную PyTorch CUDA-операцию. Для проверки нужен свободный GPU; занятая карта не отбирается у действующего занятия.
    sudo scripts/cluster.sh gpu-preflight
+
    GPU-окружения
    С Colloq поставляются два GPU-окружения: base-gpu — PyTorch и torchvision колёсами CUDA, и больше ничего, — и kaggle-gpu поверх него: набор kaggle-base в тех же закреплённых версиях плюс seaborn, OpenCV и timm. Ни один из этих образов не публикуется. На своей машине они собираются там, где стоит карта: при первом запуске с KERNEL_PRELOAD=base,base-gpu,kaggle-gpu или кнопкой «Собрать» в панели «Окружения». base-gpu занимает около 9,9 ГБ диска, kaggle-gpu — ещё около 2,8 ГБ, всего около 12,7 ГБ. В обоих torch 2.14.1 и torchvision 0.29.1 — колёса под CUDA 13.0, поэтому машине с картой нужен драйвер NVIDIA 580 или новее: на старом PyTorch импортируется, но карты не видит (torch.cuda.is_available() возвращает False). Версию драйвера и старшую CUDA, которую он поддерживает, nvidia-smi показывает в шапке. Для выпуска назовите их в --environments вместе с двумя GPU-параметрами выше. Соревнование на любом из них считает на процессоре, пока в нём не включена «Видеокарта», см. видеокарту для посылок.
    Каждая GPU-комната запрашивает один nvidia.com/gpu и RuntimeClass nvidia. Совместное использование устройства не включается автоматически. Несколько одновременных GPU-комнат требуют соответствующего числа доступных устройств либо отдельно спроектированной и проверенной политики sharing.
    Установщик сохраняет драйвер хоста, ставит закреплённые NVIDIA Container Toolkit и device plugin и проверяет реальную PyTorch CUDA-операцию. Для проверки нужен свободный GPU; занятая карта не отбирается у действующего занятия.
    sudo scripts/cluster.sh gpu-preflight
 sudo k3s kubectl get nodes \
   -o "custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
    Обычная доступность приложения не доказывает GPU-готовность. Если карта не выделяется или CUDA не работает, разберите причину до начала занятия.
    
 
    Сохраняйте старые образы
    Исторические ревизии каталога сохраняются, пока на них ссылаются комнаты. Не удаляйте соответствующие образы из registry. Резервная копия приложения содержит каталог и digest, но не слои образов.
    
diff --git a/docs/pages/ru/install.html b/docs/pages/ru/install.html
index 86881690..6d2d8715 100644
--- a/docs/pages/ru/install.html
+++ b/docs/pages/ru/install.html
@@ -1,5 +1,5 @@
 
-
    Что подготовить
    Установка на k3s рассчитана на одну Linux amd64 VM с systemd и IPv4. Нужны root-доступ, curl, Python 3, iptables (на хосте с IPv6 ещё и ip6tables) и стандартные Linux-утилиты. Для GPU-выпуска нужны Debian или Ubuntu с apt, пакет gnupg и заранее установленный рабочий драйвер NVIDIA: установщик его не заменяет. Node.js и Docker daemon на сервер устанавливать не требуется: приложение и ядра работают через k3s/containerd.
    Выделите ресурсы для k3s, приложения и каждой одновременно работающей комнаты. Число подключённых студентов само по себе не определяет нагрузку: важны код, память, файлы и число активных ядер. См. планирование ресурсов.
    
+
    Что подготовить
    Установка на k3s рассчитана на одну Linux amd64 VM с systemd и IPv4. Нужны root-доступ, curl, Python 3, iptables (на хосте с IPv6 ещё и ip6tables) и стандартные Linux-утилиты. Для GPU-выпуска нужны Debian или Ubuntu с apt, пакет gnupg и заранее установленный рабочий драйвер NVIDIA версии 580 или новее: установщик его не заменяет, а на старом драйвере torch 2.14.1 из GPU-окружений, собранный под CUDA 13.0, карты не видит. Node.js и Docker daemon на сервер устанавливать не требуется: приложение и ядра работают через k3s/containerd.
    Выделите ресурсы для k3s, приложения и каждой одновременно работающей комнаты. Число подключённых студентов само по себе не определяет нагрузку: важны код, память, файлы и число активных ядер. См. планирование ресурсов.
    
 
    Проверьте комплект выпуска
    Получите артефакты через доверенный канал. Проверьте контрольные суммы, распакуйте архив инструментов и работайте из его каталога. Манифест фиксирует исходный коммит, реальные digest образов приложения, брокера и окружений, точную версию k3s и хеши инструментов.
    sha256sum -c SHA256SUMS
 tar -xzf colloq-deploy.tar.gz
 python3 scripts/release.py validate --release release.json
diff --git a/docs/pages/ru/kubernetes.html b/docs/pages/ru/kubernetes.html
index 35e76643..68693e7f 100644
--- a/docs/pages/ru/kubernetes.html
+++ b/docs/pages/ru/kubernetes.html
@@ -1,8 +1,8 @@
 
    Кратко для платформенной команды
    Что этоColloq — совместные тетради Jupyter для занятий: веб-приложение, брокер среды выполнения и отдельный под с ядром Python для каждой комнаты. В кластер ставится Helm-чартом oci://ghcr.io/colloq-edu/charts/colloq в одно пространство имён. Открытый код, лицензия MIT.
    Что работаетDeployment приложения, одна реплика: база — SQLite. Deployment брокера, одна реплика: только он обращается к API Kubernetes. Под и Service на каждую работающую комнату. Второй под и Service комнаты — для личных тетрадей студентов. Короткоживущие поды соревнований: посылка, метрика, подготовка пакетов. Поды комнат и соревнований создаёт брокер, а не чарт.
    Что нужноОдно пространство имён с правами его администратора; Role брокера из раздела «Права»; StorageClass — ReadWriteOnce обязательно, ReadWriteMany желательно; ingress-контроллер и сертификат; зеркало реестра с образами выпуска.
    Чего не нужно никогдаcluster-admin и кластерных объектов: Namespace, PersistentVolume, ClusterRole, CRD, DaemonSet, RuntimeClass. Привилегированных контейнеров, hostPath, hostNetwork и hostPID. Root: все контейнеры работают под uid 1000. Выхода в интернет из кластера.
    СетьВходящие — только от ingress-контроллера к приложению, порт 3000. Внутри пространства имён: приложение → брокер, приложение и брокер → комнаты. Исходящие: брокер → API Kubernetes, поды → DNS кластера; по желанию приложение → модель ИИ и исходящий прокси. Коду студентов доступен только DNS, посылкам соревнований — ничего.
    ДанныеДва PVC: colloq-data — база SQLite, ключи, файлы страниц занятий, данные и посылки соревнований, снимки базы; colloq-workspace — файлы комнат.
    Резервные копииСнимки томов вашими средствами (Velero, CSI) и согласованные снимки базы, которые приложение раз в сутки само пишет в том colloq-data.
    
 
    Браузеры студентов и преподавателейIngress-контроллер кластера, TLSПриложение · Deployment colloq-app, 1 репликаБрокер · Deployment colloq-runtime, 1 реплика
    Комната A · под и ServiceКомната B · под и ServiceЛичные тетради комнатыПосылка соревнования
    Приложение — сервер на Node.js с собранным веб-интерфейсом и базой SQLite — хранит всё на двух томах и к API Kubernetes не обращается. Когда в комнату входят или запускают ячейку, приложение просит брокер поднять её под. Брокер создаёт под и Service с ClusterIP по шаблону, зашитому в его код, дожидается Jupyter и возвращает приложению адрес и токен; дальше приложение говорит с Jupyter комнаты напрямую, на порту 8888. Под комнаты живёт, пока в ней кто-то есть, и ещё 2 часа после того, как все ушли. Перезапуск и обновление приложения и брокера поды комнат переживают: новый брокер находит их по меткам, и переменные Python остаются.
    
-
    Что мы предполагаем о кластере
    Это список для вашей команды: пройдите его до установки. На каждый пункт ответ — «да, так и есть» или значение чарта, которое нужно задать.
    ЧтоМы предполагаемЕсли у вас иначе
    Версия Kubernetes1.30 или новее, совместимый с upstream: ванильный, Deckhouse и подобные. Менять память и ядра работающей комнаты без перезапуска можно с 1.33 и при праве pods/resize.Версия ниже 1.33 или право не выдаётся — runtime.inPlaceResize: false. Правила нет в Role, изменение ресурсов живой комнаты отвечает «не поддерживается», без сбоя, а новое число получит следующий под этой комнаты.
    Пространство имёнНам выдают одно пространство имён с правами его администратора, и в нём одна установка Colloq: имена объектов фиксированы (colloq-app, colloq-runtime…). Кластерных объектов чарт не создаёт.Пространство имён, его метки и квоту создаёт ваша команда. Role и RoleBinding создаёт сам чарт, а Kubernetes не даёт выдать право, которого нет у того, кто его выдаёт: ставить чарт должен тот, у кого есть все глаголы из раздела «Права».
    Pod Security и политикиДействует Pod Security Admission restricted, а Kyverno или Gatekeeper требуют requests и limits у каждого контейнера, пробы, runAsNonRoot и readOnlyRootFilesystem, запрещают privileged, hostPath и hostNetwork, пускают образы только из внутреннего реестра и только по дайджесту и требуют стандартных меток app.kubernetes.io/*.Все поды Colloq проходят restricted; подробности и исключения — в разделе «Безопасность». Свои метки добавляет commonLabels (всем объектам чарта и подам брокера), метки и аннотации только подам брокера — rooms.podLabels и rooms.podAnnotations.
    Сетевые политикиNetworkPolicy исполняются (Calico, Cilium), и пространство имён может быть закрыто по умолчанию (default-deny).Чарт объявляет каждый нужный поток, входящий и исходящий; они перечислены в разделе «Сеть». Задайте, где живут ingress-контроллер (networkPolicy.ingressController.from) и DNS кластера (networkPolicy.dns.to), и адреса API-сервера (networkPolicy.kubeApiServer).
    Ingress и TLSВход через ingress-nginx; класс и аннотации настраиваются. TLS — сертификат организации из готового Secret или от cert-manager. Пользователи приходят из корпоративной сети, через VPN или, если студенты внешние, из интернета.Другой контроллер — ingress.className и ingress.annotations с тем же смыслом: тело запроса от 256 МБ, тайм-ауты от часа, без буферизации ответов. Или ingress.enabled: false: опубликуйте Service colloq-app:3000 своим объектом и задайте config.publicUrl.
    ВходСтуденты входят по ссылке занятия, преподаватели — по личной ссылке. Перед Colloq может стоять прокси с SSO, который передаёт подписанный JWT: Teleport Application Access, oauth2-proxy или Pomerium перед вашим IdP.config.sso.*: преподаватели входят корпоративной учётной записью, студенты с ней — под своим именем; см. «Вход через корпоративный SSO». Без прокси ничего задавать не нужно.
    ХранилищеStorageClass с ReadWriteOnce есть всегда, с ReadWriteMany (CephFS) — может быть.По умолчанию чарт рассчитан на ReadWriteOnce: всё, что монтирует тома Colloq, работает на одном узле. Есть ReadWriteMany — persistence.accessMode: ReadWriteMany, и поды комнат расходятся по кластеру; см. «Хранилище».
    Без интернетаИз кластера нет интернета. Образы идут через зеркало реестра (Harbor, Nexus), пакеты PyPI — через зеркало в Nexus, оракул — через внутреннюю модель с API, совместимым с OpenAI, или выключен. Возможны исходящий HTTP-прокси и внутренний CA.global.imageRegistry — зеркало для всех образов, включая образы ядер; dependencies.indexUrl и dependencies.mirrorEgress — зеркало PyPI; config.ai.baseUrl — модель; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. См. «Образы и зеркало реестра».
    Секреты и GitOpsСекреты приходят из Vault через External Secrets, развёртывание идёт через ArgoCD.Каждый секрет берётся из готового Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. Когда они заданы, чарт не порождает при рендеринге ни одного случайного значения, и ArgoCD не видит вечного расхождения.
    Мониторинг и журналыPrometheus Operator; журналы собираются со stdout и stderr.metrics.enabled включает /metrics с токеном, metrics.serviceMonitor.enabled — ServiceMonitor; без Prometheus Operator собирайте /metrics своими средствами. config.logFormat: json — журнал по объекту JSON в строке.
    Резервные копииСнимки томов: Velero, CSI.Приложение само пишет согласованные снимки базы в том colloq-data (config.dbSnapshotHours, config.dbSnapshotKeep), так что снимок тома всегда несёт целую копию базы.
    GPUУзлы с GPU необязательны.Для GPU-комнат задаются RuntimeClass, nodeSelector и tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations; см. «Эксплуатация».
    
+
    Что мы предполагаем о кластере
    Это список для вашей команды: пройдите его до установки. На каждый пункт ответ — «да, так и есть» или значение чарта, которое нужно задать.
    ЧтоМы предполагаемЕсли у вас иначе
    Версия Kubernetes1.30 или новее, совместимый с upstream: ванильный, Deckhouse и подобные. Менять память и ядра работающей комнаты без перезапуска можно с 1.33 и при праве pods/resize.Версия ниже 1.33 или право не выдаётся — runtime.inPlaceResize: false. Правила нет в Role, изменение ресурсов живой комнаты отвечает «не поддерживается», без сбоя, а новое число получит следующий под этой комнаты.
    Пространство имёнНам выдают одно пространство имён с правами его администратора, и в нём одна установка Colloq: имена объектов фиксированы (colloq-app, colloq-runtime…). Кластерных объектов чарт не создаёт.Пространство имён, его метки и квоту создаёт ваша команда. Role и RoleBinding создаёт сам чарт, а Kubernetes не даёт выдать право, которого нет у того, кто его выдаёт: ставить чарт должен тот, у кого есть все глаголы из раздела «Права».
    Pod Security и политикиДействует Pod Security Admission restricted, а Kyverno или Gatekeeper требуют requests и limits у каждого контейнера, пробы, runAsNonRoot и readOnlyRootFilesystem, запрещают privileged, hostPath и hostNetwork, пускают образы только из внутреннего реестра и только по дайджесту и требуют стандартных меток app.kubernetes.io/*.Все поды Colloq проходят restricted; подробности и исключения — в разделе «Безопасность». Свои метки добавляет commonLabels (всем объектам чарта и подам брокера), метки и аннотации только подам брокера — rooms.podLabels и rooms.podAnnotations.
    Сетевые политикиNetworkPolicy исполняются (Calico, Cilium), и пространство имён может быть закрыто по умолчанию (default-deny).Чарт объявляет каждый нужный поток, входящий и исходящий; они перечислены в разделе «Сеть». Задайте, где живут ingress-контроллер (networkPolicy.ingressController.from) и DNS кластера (networkPolicy.dns.to), и адреса API-сервера (networkPolicy.kubeApiServer).
    Ingress и TLSВход через ingress-nginx; класс и аннотации настраиваются. TLS — сертификат организации из готового Secret или от cert-manager. Пользователи приходят из корпоративной сети, через VPN или, если студенты внешние, из интернета.Другой контроллер — ingress.className и ingress.annotations с тем же смыслом: тело запроса от 256 МБ, тайм-ауты от часа, без буферизации ответов. Или ingress.enabled: false: опубликуйте Service colloq-app:3000 своим объектом и задайте config.publicUrl.
    ВходСтуденты входят по ссылке занятия, преподаватели — по личной ссылке. Перед Colloq может стоять прокси с SSO, который передаёт подписанный JWT: Teleport Application Access, oauth2-proxy или Pomerium перед вашим IdP.config.sso.*: преподаватели входят корпоративной учётной записью, студенты с ней — под своим именем; см. «Вход через корпоративный SSO». Без прокси ничего задавать не нужно.
    ХранилищеStorageClass с ReadWriteOnce есть всегда, с ReadWriteMany (CephFS) — может быть.По умолчанию чарт рассчитан на ReadWriteOnce: всё, что монтирует тома Colloq, работает на одном узле. Есть ReadWriteMany — persistence.accessMode: ReadWriteMany, и поды комнат расходятся по кластеру; см. «Хранилище».
    Без интернетаИз кластера нет интернета. Образы идут через зеркало реестра (Harbor, Nexus), пакеты PyPI — через зеркало в Nexus, оракул — через внутреннюю модель с API, совместимым с OpenAI, или выключен. Возможны исходящий HTTP-прокси и внутренний CA.global.imageRegistry — зеркало для всех образов, включая образы ядер; dependencies.indexUrl и dependencies.mirrorEgress — зеркало PyPI; config.ai.baseUrl — модель; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. См. «Образы и зеркало реестра».
    Секреты и GitOpsСекреты приходят из Vault через External Secrets, развёртывание идёт через ArgoCD.Каждый секрет берётся из готового Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. Когда они заданы, чарт не порождает при рендеринге ни одного случайного значения, и ArgoCD не видит вечного расхождения.
    Мониторинг и журналыPrometheus Operator; журналы собираются со stdout и stderr.metrics.enabled включает /metrics с токеном, metrics.serviceMonitor.enabled — ServiceMonitor; без Prometheus Operator собирайте /metrics своими средствами. config.logFormat: json — журнал по объекту JSON в строке.
    Резервные копииСнимки томов: Velero, CSI.Приложение само пишет согласованные снимки базы в том colloq-data (config.dbSnapshotHours, config.dbSnapshotKeep), так что снимок тома всегда несёт целую копию базы.
    GPUУзлы с GPU необязательны.Для GPU-комнат задаются RuntimeClass, nodeSelector и tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations. Посылки соревнований получают GPU только через gpu.competitions.resource — ресурс device plugin, который они просят; см. «Эксплуатация».
    
 
    Права
    Права в кластере есть только у брокера: Role и RoleBinding colloq-runtime в этом пространстве имён. Ни одной ClusterRole, никакого доступа к узлам, к Secret, к Deployment и к другим пространствам имён.
    РесурсГлаголыЗачем
    podsget, list, create, deleteПоды комнат, личных тетрадей и соревнований. list нужен для переписи: перезапущенный брокер находит работающие поды по меткам и подхватывает их, а не создаёт заново.
    servicesget, list, create, deleteService с ClusterIP для каждой комнаты и её личных тетрадей — постоянное имя, по которому приложение ходит в Jupyter, — и для прокси подготовки пакетов. Удалённая насовсем комната оставляет «надгробие»: headless Service без селектора и без ClusterIP, чтобы её идентификатор нельзя было открыть снова.
    pods/resizepatchПоменять память и ядра работающей комнаты без перезапуска (Kubernetes 1.33+): патч касается только memory и cpu контейнера kernel. Необязательно: при runtime.inPlaceResize: false этого правила нет.
    persistentvolumeclaimsgetПеред соревнованием убедиться, что том с данными на месте.
    networkpolicies (networking.k8s.io)getПеред соревнованием убедиться, что политики изоляции посылок существуют. Без них посылки не принимаются, а преподаватель видит причину.
    Ни patch, ни update на поды и Service нет: живой объект брокер не правит — меняет только ресурсы комнаты через pods/resize, а всё остальное удаляет и создаёт заново, с предусловием по UID. RBAC не умеет ограничивать поля пода, поэтому шаблоны подов зашиты в код брокера, а его HTTP API принимает только идентификатор комнаты, окружение из каталога, ревизию образа, память и ядра: ни произвольного шаблона, ни hostPath, ни образа не из каталога. Остальное держат PSA restricted и ваши политики.
    ServiceAccountТокен APIКто
    colloq-appнетПриложение. К API Kubernetes не обращается вовсе.
    colloq-runtimeда: под брокера монтирует спроецированный токен, который kubelet обновляет самБрокер; связан с Role выше.
    colloq-kernelнетПоды комнат и личных тетрадей: у кода студентов нет ключей от API.
    учётная запись по умолчаниюнетПоды соревнований: токен не монтируется.
    У всех трёх ServiceAccount стоит automountServiceAccountToken: false; токен включает только под брокера.
    
-
    Безопасность
    Pod Security. Каждый под Colloq — приложение, брокер, комнаты, личные тетради, соревнования — проходит профиль restricted: runAsNonRoot, uid и gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Писать можно только в тома: emptyDir для /tmp, домашнего каталога и /dev/shm (в памяти: 64 МиБ, у GPU-комнат 1 ГиБ) и в PVC. Метку pod-security.kubernetes.io/enforce: restricted на пространство имён ставит ваша команда: чарт им не владеет.
    Движки политик.
    • requests и limits есть у каждого контейнера, включая init-контейнер и sidecar подов соревнований. У ядра комнаты requests равны limits (QoS Guaranteed): память и ядра резервируются целиком, плюс 2 ГиБ эфемерного диска.
    • Пробы. У приложения — HTTP: startupProbe и livenessProbe на /api/livez, readinessProbe на /api/readyz. У брокера — TCP 8787. У ядра комнаты и личных тетрадей — пробы готовности и живости по TCP 8888; проба живости срабатывает только после пяти минут отказов: под с restartPolicy: Never при этом завершается вместе с переменными занятия. Контейнеры посылок исполняются до конца и проб не имеют; у их выгрузчика и у прокси пакетов есть проба готовности. Если правило о пробах не подходит подам, которые работают до завершения, делайте исключение по метке app.kubernetes.io/managed-by: colloq-runtime.
    • Метки. Объекты чарта несут app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm и helm.sh/chart. Поды, которые создаёт брокер, — app.kubernetes.io/name, instance и part-of, app.kubernetes.io/managed-by: colloq-runtime и colloq.dev/role: kernel (у пода личных тетрадей ещё colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. Ключи app.kubernetes.io/* и colloq.* принадлежат чарту и брокеру, переопределить их нельзя.
    • Дайджесты. Образы приложения и брокера чарт всегда пишет как реестр/путь@sha256:… — ни тега, ни latest. Образы ядер в каталоге принимаются только в том же виде: ссылку с одним тегом не пропустят ни чарт, ни брокер.
    • Реестр. global.imageRegistry переписывает реестр каждого образа, включая ядра из каталога, выгрузчик и прокси соревнований.
    • Мутирующие вебхуки. Под комнаты брокер после создания сверяет с тем, что просил. Если вебхук изменил контейнеры, тома или настройки безопасности — внедрил sidecar, переписал imagePullPolicy, — брокер откажется от пода, и комната не запустится; в журнале брокера будут названы изменённые поля. Исключите поды с меткой app.kubernetes.io/managed-by: colloq-runtime из таких вебхуков; для Istio достаточно rooms.podLabels: {sidecar.istio.io/inject: "false"}. Размещение, которое дописывает сам кластер — PodNodeSelector, PodTolerationRestriction, приоритет, RuntimeClass, Kueue, — сверке не мешает.
    Какой код под кем работает.
    ПодОбраз и кодТокен APIТомаСеть
    Приложениеcolloq-app: сервер Colloq на Node.jsнетcolloq-data и colloq-workspaceвход от ingress-контроллера; выход к брокеру, комнатам, DNS и, по желанию, к модели ИИ и прокси
    Брокерcolloq-runtime: брокер на Node.jsданетвход от приложения; выход к API Kubernetes, комнатам, посылкам и DNS
    Комнатаcolloq-kernel окружения комнаты: Jupyter и код студентовнетсвой подкаталог colloq-workspaceвход от приложения и брокера на 8888; выход — только DNS
    Личные тетрадиcolloq-kernel: код студентов в личных тетрадях, без GPUнетподкаталог своей комнаты, по тому же путикак у комнаты
    Посылкаcolloq-kernel: тетрадь участника или метрика; colloq-runtime: выгрузчик результатанетподкаталоги colloq-data, входные — только на чтениеникакой; внутрь — только брокер к выгрузчику на 8765
    Подготовка пакетовcolloq-kernel: pip скачивает «свои пакеты» участниковнетподкаталоги colloq-dataтолько к своему прокси на 3128 и DNS
    Прокси пакетовcolloq-runtime: CONNECT-проксинетнетвход от подготовки; выход — DNS и индекс пакетов: публичный PyPI на 443 или ваше зеркало
    Изоляция комнат. У каждой комнаты свой под: свои процессы, свои emptyDir, свой токен Jupyter — брокер выводит его из секрета комнат через HMAC и отдаёт только приложению. Под монтирует лишь свой подкаталог тома colloq-workspace (subPath), файлов других комнат в нём нет. Сетевая политика пускает к порту 8888 комнаты только приложение и брокер, а из комнаты не выпускает никуда, кроме DNS: соседние комнаты, приложение и API коду студентов недоступны. Внутри комнаты участники делят ядро, файлы и пользователя Linux — так задумано: это общее рабочее место. Стандартная NetworkPolicy не гарантирует, что под не достучится до служб своего узла: проверьте это из комнаты (см. проверки после установки) и при необходимости закройте хостовыми политиками вашего CNI.
    Секреты.
    SecretЧто в нёмКто читает
    config.existingSecret или colloq-app-configКлючи — переменные приложения: OPENAI_API_KEY, SESSION_SECRET, прокси с паролем в HTTPS_PROXY или любая другая настройкаприложение
    colloq-runtime-auth, ключ runtime-tokenBearer, с которым приложение обращается к брокеруприложение и брокер
    colloq-room-secret, ключ room-secretКлюч, из которого брокер выводит токены Jupyter комнат и токены выгрузки посылоктолько брокер
    colloq-metrics, ключ metrics-tokenBearer для /metrics, если метрики включеныприложение и Prometheus
    global.imagePullSecrets.dockerconfigjson зеркала; первый из них получают и поды брокераkubelet
    Каждый можно взять из готового Secret — например, его создаёт External Secrets из Vault. Токен брокера, секрет комнат и токен метрик, если их не задали, чарт создаёт сам: при установке через helm CLI — один раз, дальше читая обратно из кластера через lookup. ArgoCD и другие инструменты, которые рендерят чарт через helm template, прочитать их не могут и создавали бы новые значения при каждой синхронизации, перезапуская комнаты, — там задайте existingSecret. Токен брокера и секрет комнат — от 43 до 128 символов [A-Za-z0-9_-] из 32 случайных байт и больше: openssl rand -hex 32. Держите секреты неизменными: новый SESSION_SECRET делает недействительными все выданные ссылки и входы, новый секрет комнат пересоздаёт поды комнат при следующем запуске, и переменные Python пропадают. Если SESSION_SECRET не задан, приложение один раз создаёт его само и хранит в томе colloq-data. Переходя с созданного чартом секрета на existingSecret, сначала перенесите его значение в новый Secret: свой чарт удалит.
    Секреты лежат и в томе colloq-data: setup-token, ключи входа преподавателей в базе, ключ модели, если его ввели в панели. Снимки этого тома храните так же закрыто, как Secret.
    
+
    Безопасность
    Pod Security. Каждый под Colloq — приложение, брокер, комнаты, личные тетради, соревнования — проходит профиль restricted: runAsNonRoot, uid и gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Писать можно только в тома: emptyDir для /tmp, домашнего каталога и /dev/shm (в памяти: 64 МиБ, у GPU-комнат 1 ГиБ, у GPU-посылки половина её памяти, не больше gpu.competitions.shmMb) и в PVC. Метку pod-security.kubernetes.io/enforce: restricted на пространство имён ставит ваша команда: чарт им не владеет.
    Движки политик.
    • requests и limits есть у каждого контейнера, включая init-контейнер и sidecar подов соревнований. У ядра комнаты requests равны limits (QoS Guaranteed): память и ядра резервируются целиком, плюс 2 ГиБ эфемерного диска.
    • Пробы. У приложения — HTTP: startupProbe и livenessProbe на /api/livez, readinessProbe на /api/readyz. У брокера — TCP 8787. У ядра комнаты и личных тетрадей — пробы готовности и живости по TCP 8888; проба живости срабатывает только после пяти минут отказов: под с restartPolicy: Never при этом завершается вместе с переменными занятия. Контейнеры посылок исполняются до конца и проб не имеют; у их выгрузчика и у прокси пакетов есть проба готовности. Если правило о пробах не подходит подам, которые работают до завершения, делайте исключение по метке app.kubernetes.io/managed-by: colloq-runtime.
    • Метки. Объекты чарта несут app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm и helm.sh/chart. Поды, которые создаёт брокер, — app.kubernetes.io/name, instance и part-of, app.kubernetes.io/managed-by: colloq-runtime и colloq.dev/role: kernel (у пода личных тетрадей ещё colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. Ключи app.kubernetes.io/* и colloq.* принадлежат чарту и брокеру, переопределить их нельзя.
    • Дайджесты. Образы приложения и брокера чарт всегда пишет как реестр/путь@sha256:… — ни тега, ни latest. Образы ядер в каталоге принимаются только в том же виде: ссылку с одним тегом не пропустят ни чарт, ни брокер.
    • Реестр. global.imageRegistry переписывает реестр каждого образа, включая ядра из каталога, выгрузчик и прокси соревнований.
    • Мутирующие вебхуки. Под комнаты брокер после создания сверяет с тем, что просил. Если вебхук изменил контейнеры, тома или настройки безопасности — внедрил sidecar, переписал imagePullPolicy, — брокер откажется от пода, и комната не запустится; в журнале брокера будут названы изменённые поля. Исключите поды с меткой app.kubernetes.io/managed-by: colloq-runtime из таких вебхуков; для Istio достаточно rooms.podLabels: {sidecar.istio.io/inject: "false"}. Размещение, которое дописывает сам кластер — PodNodeSelector, PodTolerationRestriction, приоритет, RuntimeClass, Kueue, — сверке не мешает.
    Какой код под кем работает.
    ПодОбраз и кодТокен APIТомаСеть
    Приложениеcolloq-app: сервер Colloq на Node.jsнетcolloq-data и colloq-workspaceвход от ingress-контроллера; выход к брокеру, комнатам, DNS и, по желанию, к модели ИИ и прокси
    Брокерcolloq-runtime: брокер на Node.jsданетвход от приложения; выход к API Kubernetes, комнатам, посылкам и DNS
    Комнатаcolloq-kernel окружения комнаты: Jupyter и код студентовнетсвой подкаталог colloq-workspaceвход от приложения и брокера на 8888; выход — только DNS
    Личные тетрадиcolloq-kernel: код студентов в личных тетрадях, без GPUнетподкаталог своей комнаты, по тому же путикак у комнаты
    Посылкаcolloq-kernel: тетрадь участника или метрика; colloq-runtime: выгрузчик результатанетподкаталоги colloq-data, входные — только на чтениеникакой; внутрь — только брокер к выгрузчику на 8765
    Подготовка пакетовcolloq-kernel: pip скачивает «свои пакеты» участниковнетподкаталоги colloq-dataтолько к своему прокси на 3128 и DNS
    Прокси пакетовcolloq-runtime: CONNECT-проксинетнетвход от подготовки; выход — DNS и индекс пакетов: публичный PyPI на 443 или ваше зеркало
    Изоляция комнат. У каждой комнаты свой под: свои процессы, свои emptyDir, свой токен Jupyter — брокер выводит его из секрета комнат через HMAC и отдаёт только приложению. Под монтирует лишь свой подкаталог тома colloq-workspace (subPath), файлов других комнат в нём нет. Сетевая политика пускает к порту 8888 комнаты только приложение и брокер, а из комнаты не выпускает никуда, кроме DNS: соседние комнаты, приложение и API коду студентов недоступны. Внутри комнаты участники делят ядро, файлы и пользователя Linux — так задумано: это общее рабочее место. Стандартная NetworkPolicy не гарантирует, что под не достучится до служб своего узла: проверьте это из комнаты (см. проверки после установки) и при необходимости закройте хостовыми политиками вашего CNI.
    Секреты.
    SecretЧто в нёмКто читает
    config.existingSecret или colloq-app-configКлючи — переменные приложения: OPENAI_API_KEY, SESSION_SECRET, прокси с паролем в HTTPS_PROXY или любая другая настройкаприложение
    colloq-runtime-auth, ключ runtime-tokenBearer, с которым приложение обращается к брокеруприложение и брокер
    colloq-room-secret, ключ room-secretКлюч, из которого брокер выводит токены Jupyter комнат и токены выгрузки посылоктолько брокер
    colloq-metrics, ключ metrics-tokenBearer для /metrics, если метрики включеныприложение и Prometheus
    global.imagePullSecrets.dockerconfigjson зеркала; первый из них получают и поды брокераkubelet
    Каждый можно взять из готового Secret — например, его создаёт External Secrets из Vault. Токен брокера, секрет комнат и токен метрик, если их не задали, чарт создаёт сам: при установке через helm CLI — один раз, дальше читая обратно из кластера через lookup. ArgoCD и другие инструменты, которые рендерят чарт через helm template, прочитать их не могут и создавали бы новые значения при каждой синхронизации, перезапуская комнаты, — там задайте existingSecret. Токен брокера и секрет комнат — от 43 до 128 символов [A-Za-z0-9_-] из 32 случайных байт и больше: openssl rand -hex 32. Держите секреты неизменными: новый SESSION_SECRET делает недействительными все выданные ссылки и входы, новый секрет комнат пересоздаёт поды комнат при следующем запуске, и переменные Python пропадают. Если SESSION_SECRET не задан, приложение один раз создаёт его само и хранит в томе colloq-data. Переходя с созданного чартом секрета на existingSecret, сначала перенесите его значение в новый Secret: свой чарт удалит.
    Секреты лежат и в томе colloq-data: setup-token, ключи входа преподавателей в базе, ключ модели, если его ввели в панели. Снимки этого тома храните так же закрыто, как Secret.
    
 
    Сеть
    Все потоки, которые нужны Colloq. Чарт объявляет их сетевыми политиками, входящими и исходящими, поэтому установка работает и в пространстве имён, закрытом по умолчанию.
    ОткудаКудаПортЗачем
    Ingress-контроллерПриложениеTCP 3000Весь трафик пользователей: страницы, API, WebSocket, server-sent events. Кого пускать, задаёт networkPolicy.ingressController.from (по умолчанию поды ingress-nginx из пространства имён ingress-nginx). Контроллер в сети узла приходит с адресов узлов — тогда укажите их ipBlock.
    ПриложениеБрокерTCP 8787Поднять, изменить и остановить под комнаты, запустить посылку. Запросы с Bearer-токеном.
    ПриложениеПоды комнат и личных тетрадейTCP 8888Jupyter: исполнение ячеек, терминал. С токеном комнаты.
    БрокерПоды комнат и личных тетрадейTCP 8888Проверить, что Jupyter поднялся и принимает токен.
    БрокерПоды посылок и подготовки пакетовTCP 8765Забрать результат у выгрузчика.
    БрокерAPI-сервер Kubernetes443 и 6443Создавать и удалять поды и Service. Сетевая политика видит адрес уже после DNAT, то есть сами адреса API-серверов, поэтому по умолчанию разрешён любой адрес на этих портах. Сузьте networkPolicy.kubeApiServer.cidrs до вывода kubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes. В Cilium у API-сервера своя идентичность, которую ipBlock не ловит: включите networkPolicy.kubeApiServer.ciliumEntity — чарт добавит CiliumNetworkPolicy в этом пространстве имён.
    Все поды Colloq, кроме посылокDNS кластераUDP и TCP 53Имена Service. Где DNS, задаёт networkPolicy.dns.to (по умолчанию k8s-app: kube-dns в kube-system). У комнат DNS можно убрать: rooms.network: none.
    Подготовка пакетовСвой прокси пакетовTCP 3128pip ходит только через прокси, который пускает к одному индексу пакетов.
    Прокси пакетовЗеркало PyPI или публичный PyPI443 или порт зеркалаИндекс и файлы «своих пакетов» соревнований. Политика знает адреса, а не имена: адреса и порты зеркала — в dependencies.mirrorEgress. В закрытой сети выключите общее правило для публичных адресов на 443: dependencies.publicEgress: false.
    ПриложениеМодель ИИ, исходящий прокси, GitHubпорт модели или проксиПо желанию: оракул и импорт тетрадей с GitHub. По умолчанию приложению наружу нельзя никуда: адреса и порты — в networkPolicy.app.egress.
    PrometheusПриложениеTCP 3000, /metricsПо желанию, с metrics.enabled: кого пускать, задаёт networkPolicy.monitoring.from.
    Узлы (kubelet)Зеркало реестраTCP 443Образы. Сетевыми политиками не описывается.
    Если в кластере работает NodeLocal DNSCache, запросы DNS уходят на локальный адрес узла, а не к подам kube-dns: добавьте в networkPolicy.dns.to этот адрес, например ipBlock: {cidr: 169.254.20.10/32}.
    Код студентов наружу не выходит: у подов комнат и личных тетрадей исходящих разрешений нет, кроме DNS, а rooms.network: none убирает и его. Поэтому pip install в ячейке не работает: библиотеки приходят в образах ядер, см. «Образы и зеркало реестра». Посылки соревнований не получают сети вовсе, даже DNS, и это проверяется до запуска кода: init-контейнер пробует выйти наружу и пропускает посылку, только если трижды подряд получил отказ, а если сеть ответила — посылка завершается ошибкой.
    Адрес клиента: TRUSTED_PROXIES=private. Пределы на адрес — 60 новичков в комнате за 10 минут, 20 вопросов оракулу в минуту, входы и вступления в соревнования — должны видеть адрес студента, а не ingress-контроллера. Приложение верит заголовку X-Forwarded-For только от соединений с адресов из TRUSTED_PROXIES; чарт ставит private (config.inbound.trustedProxies) — все частные диапазоны, откуда бы ни пришёл под контроллера. Это безопасно, потому что к порту 3000 приложения допускается только ingress-контроллер: сетевая политика не пускает туда больше никого, в том числе код студентов, так что подделать заголовок изнутри кластера некому. Клиентом считается самый правый адрес в заголовке, который сам не из доверенных; если доверенные все — а в корпоративной сети адреса пользователей и сами частные, — берётся самый левый, то есть тот, что записал контроллер. Поэтому контроллер должен заменять заголовок, а не дописывать свой адрес к присланному клиентом: у ingress-nginx так по умолчанию, пока use-forwarded-headers и compute-full-forwarded-for выключены. Если перед контроллером стоит балансировщик, который подменяет адрес источника, все студенты окажутся одним адресом: сохраните адрес клиента (externalTrafficPolicy: Local у Service контроллера или PROXY protocol). Если много людей приходит с одного адреса — через NAT или концентратор VPN, — перечислите его в config.inbound.sharedAddresses: пределы на адрес к нему не применяются, пределы на комнату и на человека остаются.
    ingress-nginx. Чарт ставит на свой Ingress такие аннотации (ingress.annotations; ключ со значением null убирает аннотацию):
    nginx.ingress.kubernetes.io/proxy-body-size: 256m       # данные соревнования: до 200 МБ одним запросом
 nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"  # WebSocket и server-sent events живут часами
 nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
@@ -32,7 +32,7 @@ 
    Вход через корпоративный SSO
    Если
       - cidrs: [10.20.0.15/32]              # прокси Teleport — за ключами
         ports: [443]
    Ключи можно не забирать по сети, а передать документом JWKS в config.sso.jwks: чарт смонтирует его из ConfigMap colloq-sso-jwks, исходящий доступ не нужен, но при ротации ключей в Teleport значение нужно обновить. Заголовок по умолчанию — Teleport-Jwt-Assertion; другой задаёт config.sso.header. Authorization не подходит: в нём браузер передаёт токен комнаты, и прокси, который пишет туда свой, сломал бы файлы, историю и оракул. oauth2-proxy с --pass-access-token передаёт токен в X-Forwarded-Access-Token — годится, если IdP выдаёт access token в виде JWT (так делает Keycloak); его аудитория — та, что IdP пишет в access token.
    Поля токена. По умолчанию: кто это — sub или username; имя — name, traits.name, traits.full_name, traits.display_name, username; почта — email, traits.email, traits.mail, username; роли — roles, groups, traits.groups. Другие пути задаёт config.sso.claims.*, первый непустой выигрывает. Если в токене только логин, config.sso.emailDomain превращает его в адрес, по которому узнаются преподаватели. Строка в журнале при старте говорит, откуда берутся ключи и кого Colloq добавляет; отклонённый токен попадает в журнал раз в минуту с причиной.
    Студенты без учётной записи. Через Teleport проходит только тот, у кого есть учётная запись. Если у студентов её нет, оставьте им обычный вход: Ingress на тот же Service по публичному адресу и ссылки занятий, а config.publicUrl — этот публичный адрес, из него строятся ссылки. Преподаватели при этом открывают и панель, и комнаты по адресу Teleport — ссылки для студентов панель всё равно строит от публичного адреса: проверка токена одна и та же, откуда бы ни пришёл запрос, а страница на адресе Teleport проходит проверку источника, когда агент Teleport указан в config.inbound.trustedProxies (чарт ставит private).
    Что остаётся на прокси. Токен — это пропуск до своего срока: если Colloq доступен и в обход прокси, украденный токен действует, пока не истечёт exp. Держите срок жизни токена коротким и не пишите заголовок с токеном в журналы доступа.
    
 
    Хранилище
    PVCЧто в нёмКто монтируетРазмер по умолчанию
    colloq-dataБаза SQLite (colloq.db с журналом WAL), ключ подписи ссылок, setup-token, картинки вывода, файлы страниц занятий в page-files/, данные, скрытые ответы и посылки соревнований, наборы пакетов, снимки базы в snapshots/приложение; поды соревнований — свои подкаталоги10 ГиБ; больше для соревнований с большими данными и около 0,3–0,8 ГиБ на каждый курс из 33 занятий со страницами
    colloq-workspaceФайлы комнат, <комната>/ на каждуюприложение; поды комнаты и её личных тетрадей — только её подкаталог100 ГиБ
    Класс хранения, размер и режим доступа задаются для обоих томов сразу (persistence.storageClass, persistence.accessMode) или для каждого отдельно (persistence.data.*, persistence.workspace.*); готовый PVC подключает existingClaim. С persistence.retain: true — так по умолчанию — PVC переживают и helm uninstall, и удаление приложения ArgoCD.
    ReadWriteMany — предпочтительно. С RWX у обоих томов поды комнат расходятся по узлам кластера, куда их поставит планировщик, и ёмкость установки ограничена квотой, а не одним узлом. Подходят CephFS или NFSv4, где блокировки файлов поддерживает сам протокол; писатель у SQLite один — приложение. Сама SQLite предупреждает, что на многих реализациях NFS блокировки ненадёжны: если сомневаетесь, держите том с базой на блочном хранилище (persistence.data.accessMode: ReadWriteOnce), но тогда вернётся размещение всех подов с томами на одном узле.
    ReadWriteOnce — один узел. Так чарт работает по умолчанию. Том RWO подключается к одному узлу, поэтому всё, что монтирует тома Colloq, должно работать там же: если в режиме RWO хотя бы один из томов, чарт ставит брокеру RUNTIME_COLOCATE_WITH_APP=1, и каждый под, который монтирует том, — комнаты, личные тетради, посылки, подготовка пакетов — получает обязательную podAffinity к поду приложения (метка colloq.dev/role: app, топология kubernetes.io/hostname). Сам под приложения предпочитает узел, где уже работают комнаты: после обновления новому поду нужен именно тот узел, к которому подключён том. Брокер томов не монтирует и может работать где угодно. Следствия: ёмкость всей установки — свободные ресурсы одного узла; комната сверх них не встанет, и через 20 секунд ожидания преподаватель увидит, чего не хватило, — памяти, ядер или GPU; обслуживание этого узла — перерыв для всего Colloq. Выбирайте узел с запасом, см. «Ресурсы и квоты».
    Права на файлы. Все поды пишут под uid и gid 1000. Под приложения задаёт fsGroup: 1000, и новый блочный том становится доступен группе 1000 сам. Для томов RWX драйвер CSI по умолчанию fsGroup не применяет (fsGroupPolicy: ReadWriteOnceWithFSType касается только томов RWO с файловой системой): корень такого тома должен быть доступен на запись uid 1000 со стороны хранилища.
    Квоты диска на комнату нет. Том colloq-workspace общий: студент, заполнивший его, остановит сохранение файлов у всех. Следите за заполнением PVC — метрики kubelet kubelet_volume_stats_used_bytes и kubelet_volume_stats_capacity_bytes или colloq_disk_free_bytes из /metrics, тревога на 80 %. Если StorageClass разрешает расширение томов, PVC можно увеличить.
    
-
    Образы и зеркало реестра
    ОбразЧто в нём
    ghcr.io/colloq-edu/colloq-app:vX.Y.ZПриложение.
    ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZБрокер; он же выгрузчик результатов посылок и прокси пакетов.
    ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseЯдра комнат и посылок: окружения base и kaggle-base, только CPU.
    Все образы собраны только для amd64. У выпуска и чарта один номер. В опубликованном чарте значения по умолчанию несут дайджест каждого образа выпуска: приложения и брокера — в image.app.digest и image.runtime.digest, ядер — в каталоге окружений catalog.environments. Те же значения отдельным файлом values-X.Y.Z.yaml приложены к выпуску на GitHub: по нему удобно копировать образы в зеркало и сравнивать выпуски в репозитории GitOps. Чарт из исходного дерева дайджестов не несёт и без них рендериться откажется.
    Окружение, которое выбирают в занятии, — это строка каталога: имя, образ по дайджесту, признак GPU. Своё окружение — другие библиотеки или GPU — собирается вне кластера из каталога kernel/ репозитория с KERNEL_ENV=<имя>, кладётся в ваш реестр и добавляется в catalog.environments: из панели на Kubernetes окружения не собираются. Образ base-gpu (около 9 ГБ колёс CUDA) не публикуется, GPU-окружение собирайте так же.
    Зеркало реестра. global.imageRegistry заменяет реестр в каждой ссылке на образ — у приложения, брокера, ядер из каталога, выгрузчика и прокси соревнований — и сохраняет путь и дайджест. Значение может содержать и путь: прокси-кэш ghcr.io в Harbor с проектом ghcr подходит как есть, harbor.corp.example/ghcr даёт harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. Значит, в зеркале образы должны лежать по тем же путям. Секреты для скачивания — global.imagePullSecrets: их получают поды приложения и брокера, а первый из них — и все поды, которые создаёт брокер.
    Без интернета. На машине с доступом к ghcr.io:
    # чарт выпуска
+
    Образы и зеркало реестра
    ОбразЧто в нём
    ghcr.io/colloq-edu/colloq-app:vX.Y.ZПриложение.
    ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZБрокер; он же выгрузчик результатов посылок и прокси пакетов.
    ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseЯдра комнат и посылок: окружения base и kaggle-base, только CPU.
    Все образы собраны только для amd64. У выпуска и чарта один номер. В опубликованном чарте значения по умолчанию несут дайджест каждого образа выпуска: приложения и брокера — в image.app.digest и image.runtime.digest, ядер — в каталоге окружений catalog.environments. Те же значения отдельным файлом values-X.Y.Z.yaml приложены к выпуску на GitHub: по нему удобно копировать образы в зеркало и сравнивать выпуски в репозитории GitOps. Чарт из исходного дерева дайджестов не несёт и без них рендериться откажется.
    Окружение, которое выбирают в занятии, — это строка каталога: имя, образ по дайджесту, признак GPU. Своё окружение — другие библиотеки или GPU — собирается вне кластера из каталога kernel/ репозитория с KERNEL_ENV=<имя>, кладётся в ваш реестр и добавляется в catalog.environments: из панели на Kubernetes окружения не собираются. GPU-образы — base-gpu (torch 2.14.1 и torchvision колёсами под CUDA 13.0, образ около 9,9 ГБ) и kaggle-gpu поверх него (набор kaggle-base плюс OpenCV и timm, ещё около 2,8 ГБ, всего около 12,7 ГБ) — не публикуются; собирайте их так же, сначала родителя. Узлам, где они работают с картой, нужен драйвер NVIDIA 580 или новее. Каждому поду GPU-образа, в том числе подам метрики и подготовки пакетов на узлах студентов, даётся gpu.competitions.pullSeconds (900 секунд), чтобы стянуть образ на узел, где его ещё не было, прежде чем пойдёт собственное время задания; чтобы первое занятие этого не ждало, стяните GPU-образы на эти узлы заранее, например DaemonSet’ом, у которого init-контейнеры — эти образы, каждый с командой true.
    Зеркало реестра. global.imageRegistry заменяет реестр в каждой ссылке на образ — у приложения, брокера, ядер из каталога, выгрузчика и прокси соревнований — и сохраняет путь и дайджест. Значение может содержать и путь: прокси-кэш ghcr.io в Harbor с проектом ghcr подходит как есть, harbor.corp.example/ghcr даёт harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. Значит, в зеркале образы должны лежать по тем же путям. Секреты для скачивания — global.imagePullSecrets: их получают поды приложения и брокера, а первый из них — и все поды, которые создаёт брокер.
    Без интернета. На машине с доступом к ghcr.io:
    # чарт выпуска
 helm pull oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z
 
 # все образы, на которые он ссылается, включая ядра из каталога
@@ -156,7 +156,7 @@ 
    Установка
    Пространство имён с
         print(host, port, "ОТКРЫТО: проверьте сетевые политики")
     except OSError as error:
         print(host, port, "закрыто:", error)
  • Если нужны соревнования — пробная посылка до конца; строки лидерборда должны появляться сами, без перезагрузки страницы.
  • До первой группы — восстановление из резервной копии в отдельное пространство имён.
    • 
-
    Значения чарта
    Главные значения и переменные, которые они задают. Полный список с пояснением к каждому — в values.yaml чарта: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Остальные настройки приложения из таблицы «Настройки» руководства по VM передаются в config.extraEnv или ключами Secret из config.existingSecret.
    ЗначениеПеременная и что задаётПо умолчанию
    Адрес и вход
    ingress.hostИмя хоста; обязательно, если Ingress включён.—
    config.publicUrlPUBLIC_URL — внешний адрес, из которого строятся ссылки.https://<ingress.host>
    ingress.className, ingress.annotationsКласс контроллера и аннотации.nginx, аннотации из раздела «Сеть»
    ingress.tls.secretName, ingress.tls.certManager.*Secret с сертификатом или выпускающий центр cert-manager.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — чьему X-Forwarded-For верить.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — адреса NAT и VPN, к которым не применяются пределы на адрес.пусто
    config.inbound.hstsHSTS — заголовок Strict-Transport-Security от приложения.true
    Образы и каталог
    global.imageRegistryРеестр, можно с путём, вместо ghcr.io для всех образов, включая каталог ядер.пусто
    global.imagePullSecretsSecret для скачивания образов; первый из них получают поды брокера (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestДайджесты образов приложения и брокера.дайджесты выпуска
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — каталог окружений: имя, образ по дайджесту, GPU, несколько ревизий одного имени с одной current: true.base и kaggle-base выпуска
    commonLabelsМетки всех объектов чарта и всех подов брокера (входят в RUNTIME_POD_LABELS).—
    Секреты
    config.existingSecretГотовый Secret, чьи ключи становятся переменными приложения.—
    secrets.runtimeToken.*, secrets.roomSecret.*Токен брокера и секрет комнат: existingSecret и existingSecretKey или value.создаются чартом один раз (только helm CLI)
    Хранилище
    persistence.accessModeРежим доступа томов. Если хотя бы один том ReadWriteOnce, брокер получает RUNTIME_COLOCATE_WITH_APP=1: все поды с томами — на узле приложения.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeКласс хранения и размеры томов; existingClaim — готовый PVC.класс кластера; 10Gi и 100Gi
    persistence.retainОставлять PVC при удалении релиза и приложения ArgoCD.true
    Комнаты
    rooms.memoryRUNTIME_KERNEL_MEMORY — память комнаты, если в занятии не задана своя.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — потолок памяти комнаты. Задайте явно: без него брокер берёт память своего узла за вычетом 1 ГиБ.память узла брокера − 1 ГиБ
    rooms.cpuRUNTIME_KERNEL_CPU — ядра комнаты, если в занятии не заданы свои.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — место под /tmp и домашний каталог комнаты.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — с none у комнат нет и DNS.только DNS
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — где работают поды комнат и личных тетрадей; поды соревнований их не получают.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — класс приоритета всех подов, которые создаёт брокер.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — метки и аннотации всех подов брокера; к меткам чарт сам добавляет app.kubernetes.io/name, instance и part-of.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — с false ресурсы живой комнаты не меняются, а правила pods/resize нет в Role.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — RuntimeClass GPU-комнат; пусто — без него.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — добавляются GPU-комнатам.—
    Сеть и исходящие
    networkPolicy.ingressController.fromКто допускается к порту 3000 приложения.ingress-nginx
    networkPolicy.dns.toГде DNS кластера.kube-dns в kube-system
    networkPolicy.kubeApiServer.*Адреса и порты API-сервера для брокера; ciliumEntity — для Cilium.любой адрес на 443 и 6443
    networkPolicy.app.egressКуда приложению можно наружу: модель, прокси, GitHub.никуда
    networkPolicy.monitoring.fromКто собирает /metrics.Prometheus в monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — исходящий прокси приложения. Имена Service (.svc, имена без точек) и частные адреса идут мимо него сами.—
    outbound.extraCa.pem или .existingConfigMapNODE_EXTRA_CA_CERTS — сертификаты внутреннего CA в формате PEM для приложения и подготовки пакетов.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — зеркало PyPI для «своих пакетов» соревнований: https, без учётных данных.публичный PyPI
    dependencies.mirrorEgress, dependencies.publicEgressАдреса и порты зеркала для прокси пакетов; общее правило для публичных адресов на 443.—; true
    Занятия и оракул
    config.timezoneTZ — часовой пояс: границы дня для дневных норм, даты.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — язык интерфейса, пока владелец не выберет его в панели.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — предел одного файла и всех загрузок в комнату.50 и 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — модель оракула; для модели внутри сети — vllm, ollama или custom. Ключ лучше держать в config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvДругие настройки приложения: NAME: значение.—
    Наблюдение, копии, ресурсы
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — адрес /metrics за Bearer-токеном и ServiceMonitor, который берёт тот же токен.выключено
    config.logFormatLOG_FORMAT — с json каждая строка журнала — объект JSON.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — как часто писать согласованный снимок базы в /data/snapshots и сколько хранить.24 ч и 7
    app.resources, runtime.resourcesЗапросы и пределы приложения и брокера.см. «Ресурсы и квоты»
    
+
    Значения чарта
    Главные значения и переменные, которые они задают. Полный список с пояснением к каждому — в values.yaml чарта: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Остальные настройки приложения из таблицы «Настройки» руководства по VM передаются в config.extraEnv или ключами Secret из config.existingSecret.
    ЗначениеПеременная и что задаётПо умолчанию
    Адрес и вход
    ingress.hostИмя хоста; обязательно, если Ingress включён.—
    config.publicUrlPUBLIC_URL — внешний адрес, из которого строятся ссылки.https://<ingress.host>
    ingress.className, ingress.annotationsКласс контроллера и аннотации.nginx, аннотации из раздела «Сеть»
    ingress.tls.secretName, ingress.tls.certManager.*Secret с сертификатом или выпускающий центр cert-manager.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — чьему X-Forwarded-For верить.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — адреса NAT и VPN, к которым не применяются пределы на адрес.пусто
    config.inbound.hstsHSTS — заголовок Strict-Transport-Security от приложения.true
    Образы и каталог
    global.imageRegistryРеестр, можно с путём, вместо ghcr.io для всех образов, включая каталог ядер.пусто
    global.imagePullSecretsSecret для скачивания образов; первый из них получают поды брокера (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestДайджесты образов приложения и брокера.дайджесты выпуска
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — каталог окружений: имя, образ по дайджесту, GPU, несколько ревизий одного имени с одной current: true.base и kaggle-base выпуска
    commonLabelsМетки всех объектов чарта и всех подов брокера (входят в RUNTIME_POD_LABELS).—
    Секреты
    config.existingSecretГотовый Secret, чьи ключи становятся переменными приложения.—
    secrets.runtimeToken.*, secrets.roomSecret.*Токен брокера и секрет комнат: existingSecret и existingSecretKey или value.создаются чартом один раз (только helm CLI)
    Хранилище
    persistence.accessModeРежим доступа томов. Если хотя бы один том ReadWriteOnce, брокер получает RUNTIME_COLOCATE_WITH_APP=1: все поды с томами — на узле приложения.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeКласс хранения и размеры томов; existingClaim — готовый PVC.класс кластера; 10Gi и 100Gi
    persistence.retainОставлять PVC при удалении релиза и приложения ArgoCD.true
    Комнаты
    rooms.memoryRUNTIME_KERNEL_MEMORY — память комнаты, если в занятии не задана своя.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — потолок памяти комнаты. Задайте явно: без него брокер берёт память своего узла за вычетом 1 ГиБ.память узла брокера − 1 ГиБ
    rooms.cpuRUNTIME_KERNEL_CPU — ядра комнаты, если в занятии не заданы свои.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — место под /tmp и домашний каталог комнаты.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — с none у комнат нет и DNS.только DNS
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — где работают поды комнат, личных тетрадей и соревнований: посылок, метрики и подготовки пакетов (но не её прокси пакетов). Поду GPU-посылки сверху добавляются gpu.nodeSelector и gpu.tolerations, так что его узел должен подходить под оба.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — класс приоритета всех подов, которые создаёт брокер.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — метки и аннотации всех подов брокера; к меткам чарт сам добавляет app.kubernetes.io/name, instance и part-of.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — с false ресурсы живой комнаты не меняются, а правила pods/resize нет в Role.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — RuntimeClass GPU-комнат и GPU-посылок; пусто — без него.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — добавляются GPU-комнатам и GPU-посылкам.—
    gpu.competitions.resource, gpu.competitions.countRUNTIME_COMPETITION_GPU_RESOURCE, RUNTIME_COMPETITION_GPU_COUNT — что просит под тетради соревнования с включённой «Видеокартой»: имя ресурса device plugin, например nvidia.com/gpu или nvidia.com/mig-1g.10gb, и сколько его единиц, от 1 до 8. nvidia.com/gpu просят и GPU-комнаты: с ним посылки занимают карты комнат, см. узлы с GPU. Пусто — посылки GPU не получают, а такое соревнование отказывает в загрузке.пусто, 1
    gpu.competitions.slotsRUNTIME_COMPETITION_GPU_SLOTS — сколько GPU-посылок этой установки кластер исполняет одновременно, от 1 до 256: единицы, отведённые посылкам, делённые на count. Очередь приложения держит у брокера столько и пропускает процессорную работу мимо остальных, см. узлы с GPU.пусто: по одной
    gpu.competitions.shmMbRUNTIME_COMPETITION_GPU_SHM_MB — потолок /dev/shm в памяти у GPU-посылки, МиБ, от 0 до 65536; больше половины памяти посылки он не бывает. 0 — без него, так нужно при разделении MPS.1024
    gpu.competitions.pullSecondsRUNTIME_COMPETITION_GPU_PULL_SECONDS — сколько любой под образа GPU-окружения из каталога может тянуть образ на свой узел, прежде чем пойдёт собственное время задания, от 0 до 3600 секунд: посылка, метрика и подготовка пакетов одинаково, с картой и без.900
    Сеть и исходящие
    networkPolicy.ingressController.fromКто допускается к порту 3000 приложения.ingress-nginx
    networkPolicy.dns.toГде DNS кластера.kube-dns в kube-system
    networkPolicy.kubeApiServer.*Адреса и порты API-сервера для брокера; ciliumEntity — для Cilium.любой адрес на 443 и 6443
    networkPolicy.app.egressКуда приложению можно наружу: модель, прокси, GitHub.никуда
    networkPolicy.monitoring.fromКто собирает /metrics.Prometheus в monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — исходящий прокси приложения. Имена Service (.svc, имена без точек) и частные адреса идут мимо него сами.—
    outbound.extraCa.pem или .existingConfigMapNODE_EXTRA_CA_CERTS — сертификаты внутреннего CA в формате PEM для приложения и подготовки пакетов.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — зеркало PyPI для «своих пакетов» соревнований: https, без учётных данных.публичный PyPI
    dependencies.mirrorEgress, dependencies.publicEgressАдреса и порты зеркала для прокси пакетов; общее правило для публичных адресов на 443.—; true
    Занятия и оракул
    config.timezoneTZ — часовой пояс: границы дня для дневных норм, даты.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — язык интерфейса, пока владелец не выберет его в панели.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — предел одного файла и всех загрузок в комнату.50 и 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — модель оракула; для модели внутри сети — vllm, ollama или custom. Ключ лучше держать в config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvДругие настройки приложения: NAME: значение.—
    Наблюдение, копии, ресурсы
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — адрес /metrics за Bearer-токеном и ServiceMonitor, который берёт тот же токен.выключено
    config.logFormatLOG_FORMAT — с json каждая строка журнала — объект JSON.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — как часто писать согласованный снимок базы в /data/snapshots и сколько хранить.24 ч и 7
    app.resources, runtime.resourcesЗапросы и пределы приложения и брокера.см. «Ресурсы и квоты»
    
 
    Ресурсы и квоты
    Цифры выведены из пределов в коде и из того же расчёта, что для одной VM; здесь они переведены в requests и limits подов. Главное отличие от VM: под комнаты резервирует память и ядра целиком, а не делит их с соседями.
    ПодСколькоCPU: запрос / пределПамять: запрос / пределЭфемерный диск
    Приложение1250m / 2512Mi / 2Gi—
    Брокер1100m / 1128Mi / 512Mi—
    Комнатана каждую работающую2 / 24Gi / 4Gi2Gi
    Личные тетрадина комнату, где ими пользуютсякак у комнаты или своё число класса, запрос равен пределу; нужно около 250m на студентакак у комнаты или «Память на класс»; нужно 1–1,5 ГиБ на студента2Gi
    Посылкана слотядра посылки + 100m / + 500mпамять посылки + 64Mi / + 256Mi128Mi / 320Mi
    Подготовка пакетовпока идёт1100m / 1500m, прокси 100m / 500m2112Mi / 2304Mi, прокси 64Mi / 128Miдо 320Mi
    Минимальная квота для одного занятия за раз (комната 2 ядра и 4 ГиБ), в ключах ResourceQuota — requests.cpu, requests.memory, limits.cpu, limits.memory, pods:
    СценарийЗапрос CPUЗапрос памятиПредел CPUПредел памятиПодов
    (a) 30 студентов, одна общая тетрадь2,354,6 ГиБ56,5 ГиБ3
    (b) 30 студентов, у каждого личная тетрадь (1,5 ГиБ и 0,25 ядра на студента)9,8549,6 ГиБ12,551,5 ГиБ4
    (c) 30 студентов и соревнование, 7 слотов по 2 ядра и 2 ГиБ17,0519,1 ГиБ22,522,25 ГиБ10
    (e) 100 человек в лекционной комнате2,354,6 ГиБ56,5 ГиБ3
    Комната держит свой под, пока в ней кто-то есть, и ещё 2 часа после того, как все ушли, поэтому занятия подряд накладываются: на каждое следующее занятие в пределах двух часов добавьте ещё комнату. Слоты соревнований — одна очередь на всю установку. Пример квоты для сценария (c) с запасом на комнату следующего занятия и на подготовку пакетов:
    apiVersion: v1
 kind: ResourceQuota
 metadata:
@@ -172,7 +172,7 @@ 
    Ресурсы и квоты
    Цифры выведены
     pods: "25"
     services: "100"
     persistentvolumeclaims: "2"
-    requests.storage: 150Gi
    • Под сверх квоты API не создаст, и комната или посылка получит отказ. Держите квоту выше суммы, а число слотов — в её пределах: «Исполнителей одновременно» во вкладке «Ресурсы» или COMPETITION_SLOTS в config.extraEnv. Автоматическое число считается по ядрам и памяти узла, где работает приложение, и о вашей квоте ничего не знает.
    • Service копятся: каждая комната, удалённая насовсем, оставляет headless Service без ClusterIP, поэтому предел на число Service ставьте с запасом.
    • LimitRange с максимумом на контейнер должен пропускать комнату (2 ядра и 4 ГиБ, GPU-комнату — с её памятью) и самую большую посылку, которую вы разрешите в соревнованиях: по умолчанию 2 ядра и 4 ГиБ.
    • С ReadWriteOnce вся квота должна поместиться на один узел: для сценария (c) — узел, где свободно не меньше 24 vCPU и 32 ГиБ, для (b) — не меньше 16 vCPU и 64 ГиБ.
    Замер приложения (30.09.2026, VM 21 vCPU / 49 ГБ, nginx перед приложением, make load): 500 человек вошли в одну комнату за минуту без единого отказа, вход — 7 мс в середине распределения. Процесс приложения держал 243 МБ памяти и 20 % одного ядра в покое и 50 % ядра, когда двадцать человек печатали одновременно; правка доходила до остальных за 0,17 с (p95). На 300 человек — 198 МБ и те же доли ядра. Замер сделан на VM, а не в кластере, но процесс тот же: запрос чарта в 512 МиБ вдвое больше замеренного, а пределы в 2 ядра и 2 ГиБ оставляют большой запас.
    
+    requests.storage: 150Gi
    • Под сверх квоты API не создаст, и комната или посылка получит отказ. Держите квоту выше суммы, а число слотов — в её пределах: «Исполнителей одновременно» во вкладке «Ресурсы» или COMPETITION_SLOTS в config.extraEnv. Автоматическое число считается по ядрам и памяти узла, где работает приложение, и о вашей квоте ничего не знает. GPU-посылка с таким отказом вместо этого возвращается в очередь и пробуется снова. Квота, где назван GPU-ресурс, requests.nvidia.com/gpu, ограничивает GPU-комнаты и GPU-посылки вместе: держите gpu.competitions.slots × gpu.competitions.count в её пределах за вычетом того, что держат GPU-комнаты.
    • Service копятся: каждая комната, удалённая насовсем, оставляет headless Service без ClusterIP, поэтому предел на число Service ставьте с запасом.
    • LimitRange с максимумом на контейнер должен пропускать комнату (2 ядра и 4 ГиБ, GPU-комнату — с её памятью) и самую большую посылку, которую вы разрешите в соревнованиях: по умолчанию 2 ядра и 4 ГиБ.
    • С ReadWriteOnce вся квота должна поместиться на один узел: для сценария (c) — узел, где свободно не меньше 24 vCPU и 32 ГиБ, для (b) — не меньше 16 vCPU и 64 ГиБ.
    Замер приложения (30.09.2026, VM 21 vCPU / 49 ГБ, nginx перед приложением, make load): 500 человек вошли в одну комнату за минуту без единого отказа, вход — 7 мс в середине распределения. Процесс приложения держал 243 МБ памяти и 20 % одного ядра в покое и 50 % ядра, когда двадцать человек печатали одновременно; правка доходила до остальных за 0,17 с (p95). На 300 человек — 198 МБ и те же доли ядра. Замер сделан на VM, а не в кластере, но процесс тот же: запрос чарта в 512 МиБ вдвое больше замеренного, а пределы в 2 ядра и 2 ГиБ оставляют большой запас.
    
 
    Эксплуатация
    Обновление. helm upgrade colloq oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z -n colloq -f colloq-values.yaml или новый targetRevision в ArgoCD. Приложение и брокер обновляются стратегией Recreate: старый под останавливается до старта нового, потому что писатель у SQLite один. Перерыв — от секунд до минуты; на старт с миграцией базы отведено до 10 минут. Открытые комнаты переподключаются сами. Поды комнат обновление переживают: брокер при остановке их не трогает, новый подхватывает, и переменные Python остаются. Настройки размещения — rooms.nodeSelector, rooms.tolerations, класс приоритета, режим хранилища — достаются следующему поду комнаты, а работающий остаётся, где стоял. И всё же обновляйтесь между занятиями.
    Ревизии ядер. Комната остаётся на той ревизии образа ядра, с которой начала, и запускает только её. При helm upgrade чарт сам оставляет в каталоге ревизии прежнего выпуска с current: false (не больше catalog.maxRetainedPerEnvironment, по умолчанию трёх на окружение): он читает их из живого ConfigMap colloq-catalog, и комнаты, начатые раньше, продолжают на своём Python. ArgoCD рендерит без доступа к кластеру и прежний каталог не видит: там комната, чьей ревизии больше нет, при следующем запуске переходит на текущую ревизию своего окружения, а в журнале приложения остаётся строка об этом. Чтобы и под ArgoCD держать прежние ревизии, перечислите их в catalog.environments с current: false. Образы прежних ревизий держите в зеркале, пока ими пользуются комнаты.
    Версия данных. Если новый выпуск поднимает версию схемы базы, приложение перед миграцией само пишет снимок /data/snapshots/pre-schema-<было>-to-<стало>-<время>.db; такие снимки сами не удаляются. Прежний выпуск на базе с более новой схемой не запустится: в журнале будет «colloq.db … was written by a newer Colloq», под уйдёт в CrashLoopBackOff. Поэтому откат в таком случае — это прежний чарт вместе с восстановлением базы: остановите приложение, из служебного пода положите снимок на место colloq.db и удалите colloq.db-wal и colloq.db-shm, затем установите прежнюю версию чарта. Всё, что сделано после обновления, пропадёт. COLLOQ_ALLOW_SCHEMA_DOWNGRADE=1 в config.extraEnv запускает старый выпуск на новой базе без восстановления — только если вы знаете, что миграция для него безвредна.
    kubectl -n colloq scale deploy/colloq-app --replicas=0
 kubectl -n colloq apply -f - <<'EOF'
 apiVersion: v1
@@ -205,5 +205,5 @@ 
    Эксплуатация
    Обновление
 kubectl -n colloq exec colloq-maintenance -- sh -c \
   'cp /data/snapshots/pre-schema-….db /data/colloq.db && rm -f /data/colloq.db-wal /data/colloq.db-shm'
 kubectl -n colloq delete pod colloq-maintenance
-helm rollback colloq <ревизия до обновления> -n colloq
    Резервные копии. Снимайте оба PVC вашими средствами — Velero со снимками CSI или снимками хранилища — и Secret пространства имён, если они не из Vault: тогда копией секретов служит Vault. Снимок живого тома SQLite — это снимок как при внезапном отключении: база по нему обычно восстанавливается, но полагаться на это не стоит. Поэтому приложение каждые config.dbSnapshotHours часов (в чарте — 24) само пишет согласованный снимок /data/snapshots/colloq-<время>.db и хранит последние config.dbSnapshotKeep (7): в любом снимке тома есть целая копия базы не старше суток. Тома снимаются по отдельности, так что файлы комнат и база в копии могут расходиться на минуты. Копии тома colloq-data содержат ключи входа преподавателей, setup-token и ключ модели, если его вводили в панели, — храните их как секреты. Восстановление в отдельное пространство имён проверяйте хотя бы раз в семестр. Команды colloq-host backup и restore — для VM, в кластере они не используются.
    Наблюдение. Проба живости — /api/livez, готовности — /api/readyz: она проверяет только базу и рабочие файлы, так что сбой брокера или API не выводит сайт из ротации, а трудности с ядрами видны там, где запускают код, и в /api/health. С metrics.enabled адрес /metrics отдаёт метрики в текстовом формате Prometheus тому, кто пришёл с Authorization: Bearer <токен>; без настройки он отвечает 404. Для тревог пригодятся свободное место на томах (colloq_disk_free_bytes), время последнего снимка базы (colloq_db_snapshot_last_timestamp_seconds), сбои сохранения тетрадей и запуска ядер (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Журнал — stdout и stderr; с config.logFormat: json каждая строка — объект JSON с полями time, level, msg и контекстом. В журнале есть адреса комнат, а адрес комнаты — это ссылка для входа: доступ к журналу равен доступу к комнатам. Преподавателям доступен GET /api/instance/operations: возраст очереди, повторы, сбои сохранения тетрадей, резервы памяти, свободный диск.
    Узлы с GPU. GPU-комната просит nvidia.com/gpu: 1 и RuntimeClass из gpu.runtimeClassName (по умолчанию nvidia, пусто — без него); узлы выбирают gpu.nodeSelector и gpu.tolerations. Драйвер, device plugin или GPU Operator и сам RuntimeClass — кластерные объекты, их ставит ваша команда. Одна карта на комнату, без разделения, если ваш device plugin не делит карты сам. Личные тетради и посылки соревнований GPU не получают никогда. Образ GPU-окружения соберите сами, см. «Образы и зеркало реестра».
    Обслуживание узлов. Поды комнат — одиночные поды без контроллера и с emptyDir: kubectl drain удалит их только с --force --delete-emptydir-data, а cluster-autoscaler не освободит узел с ними, пока вы не добавите аннотацию cluster-autoscaler.kubernetes.io/safe-to-evict: "true" через rooms.podAnnotations. Удалённый под комната поднимет снова при следующем запуске: тетради и файлы на месте, переменные Python пропадают. С ReadWriteOnce обслуживание узла приложения — перерыв для всей установки.
    Удаление. helm uninstall убирает объекты чарта, кроме PVC — с persistence.retain: true они остаются, — и не трогает поды и Service, которые создал брокер: они не принадлежат Helm. Удалите их по метке: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
-
    Честные ограничения
    • Одна реплика приложения. База — SQLite, писатель один: отказоустойчивости нет, обновление и отказ узла прерывают работу на время перезапуска пода. Чтобы расти, ставят отдельные установки, по одной на пространство имён, — например, по одной на подразделение.
    • SSO — только через прокси. Своего входа через SSO, LDAP и второго фактора у Colloq нет: его отдают прокси, который подписывает JWT, — см. «Вход через корпоративный SSO». Без такого прокси владелец и преподаватели входят по личным ссылкам; панель можно закрыть на ingress-контроллере — например, oauth2-proxy с корпоративным IdP через аннотации внешней аутентификации ingress-nginx на отдельном Ingress для путей /admin и /api/admin. Участники соревнований и с прокси входят по своему ключу.
    • Ссылка комнаты — общая ссылка. Кто её получил, тот войдёт и по умолчанию сможет запускать код; отозвать её нельзя. Для индивидуально оцениваемых и конфиденциальных работ комнаты не подходят: участники делят ядро, файлы и пользователя Linux.
    • ReadWriteOnce — это один узел. Вся установка помещается на один узел, и его обслуживание — перерыв для всех.
    • У кода студентов нет интернета. pip install в ячейке не работает; библиотеки — только в образах ядер из каталога, которые собираются вне кластера.
    • Нет квоты диска на комнату: том colloq-workspace общий.
    • Сетевые политики — не вся изоляция. Стандартная NetworkPolicy не гарантирует, что под не достучится до служб своего узла, а поды делят ядро Linux узла.
    • Только amd64.
    • Ядра — на Debian 12. Сканер найдёт в них критические находки, которые Debian не исправил; см. «Образы и зеркало реестра».
    • Путь новый. Чарт появился в этом выпуске: проверьте установку в непроизводственном пространстве имён, прежде чем вести в нём занятия.
    
+helm rollback colloq <ревизия до обновления> -n colloq
    Резервные копии. Снимайте оба PVC вашими средствами — Velero со снимками CSI или снимками хранилища — и Secret пространства имён, если они не из Vault: тогда копией секретов служит Vault. Снимок живого тома SQLite — это снимок как при внезапном отключении: база по нему обычно восстанавливается, но полагаться на это не стоит. Поэтому приложение каждые config.dbSnapshotHours часов (в чарте — 24) само пишет согласованный снимок /data/snapshots/colloq-<время>.db и хранит последние config.dbSnapshotKeep (7): в любом снимке тома есть целая копия базы не старше суток. Тома снимаются по отдельности, так что файлы комнат и база в копии могут расходиться на минуты. Копии тома colloq-data содержат ключи входа преподавателей, setup-token и ключ модели, если его вводили в панели, — храните их как секреты. Восстановление в отдельное пространство имён проверяйте хотя бы раз в семестр. Команды colloq-host backup и restore — для VM, в кластере они не используются.
    Наблюдение. Проба живости — /api/livez, готовности — /api/readyz: она проверяет только базу и рабочие файлы, так что сбой брокера или API не выводит сайт из ротации, а трудности с ядрами видны там, где запускают код, и в /api/health. С metrics.enabled адрес /metrics отдаёт метрики в текстовом формате Prometheus тому, кто пришёл с Authorization: Bearer <токен>; без настройки он отвечает 404. Для тревог пригодятся свободное место на томах (colloq_disk_free_bytes), время последнего снимка базы (colloq_db_snapshot_last_timestamp_seconds), сбои сохранения тетрадей и запуска ядер (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Журнал — stdout и stderr; с config.logFormat: json каждая строка — объект JSON с полями time, level, msg и контекстом. В журнале есть адреса комнат, а адрес комнаты — это ссылка для входа: доступ к журналу равен доступу к комнатам. Преподавателям доступен GET /api/instance/operations: возраст очереди, повторы, сбои сохранения тетрадей, резервы памяти, свободный диск.
    Узлы с GPU. GPU-комната просит nvidia.com/gpu: 1 и RuntimeClass из gpu.runtimeClassName (по умолчанию nvidia, пусто — без него); узлы выбирают gpu.nodeSelector и gpu.tolerations. Драйвер, device plugin или GPU Operator и сам RuntimeClass — кластерные объекты, их ставит ваша команда. Поставляемым GPU-окружениям нужен драйвер 580 или новее: base-gpu и kaggle-gpu несут torch 2.14.1, собранный под CUDA 13.0, и на старом драйвере он импортируется, но карты не видит. Одна карта на комнату, без разделения, если ваш device plugin не делит карты сам. Личные тетради GPU не получают никогда. Посылка соревнования получает его, только если в соревновании включена «Видеокарта» и задано gpu.competitions.resource: под её тетради, но никогда не метрики, просит gpu.competitions.count единиц этого ресурса, с nodeSelector и tolerations комнат и поверх них RuntimeClass, nodeSelector и tolerations GPU-комнат, и с /dev/shm в памяти — в половину её памяти, не больше gpu.competitions.shmMb (1024 МиБ). Что такое одна единица — целая карта, реплика time-slicing или MPS, часть MIG, — решает настройка вашего device plugin, и видеопамять здесь Colloq сам не ограничивает. При разделении MPS поставьте gpu.competitions.shmMb: 0: device plugin монтирует в /dev/shm общую память своего демона MPS, наш том на том же месте её закрыл бы, и CUDA в посылке не запустилась бы. GPU-комнаты держите подальше от узлов MPS — это и дают переименованные реплики (ниже): у комнаты свой /dev/shm на 1 ГиБ. Без этого значения соревнование с включённой «Видеокартой» отказывает в загрузке и называет причину. Сколько GPU-посылок одновременно уходит в кластер, задаёт gpu.competitions.slots, по одной, пока оно не задано: очередь приложения держит у брокера столько, а процессорные посылки проходят мимо остальных; если же кластер вернул посылку из-за нехватки единицы, очередь не берёт GPU-посылок, пока не подойдёт её повтор или не закончится одна из GPU-посылок. GPU-посылка, которую планировщик не может разместить или чей под отклонил API-сервер — исчерпана ResourceQuota, в кластере нет нужного RuntimeClass, запретила политика допуска, — возвращается в очередь и пробуется снова через время от нескольких секунд до минуты, а преподаватель видит причину. С nvidia.com/gpu — тем ресурсом, что просят GPU-комнаты, — посылки берут единицы с карт комнат: посылка держит свою до предела времени соревнования, а GPU-комнату, которой не хватило свободной карты, через 20 секунд не запускают. Если на занятиях работают GPU-комнаты, дайте посылкам собственный ресурс, чтобы те и другие никогда не занимали одни карты: реплики time-slicing, переименованные в nvidia.com/gpu.shared (renameByDefault: true), на узлах, размеченных под эту настройку device plugin, или профиль MIG вроде nvidia.com/mig-1g.10gb. Образы GPU-окружений соберите сами, см. «Образы и зеркало реестра».
    Обслуживание узлов. Поды комнат — одиночные поды без контроллера и с emptyDir: kubectl drain удалит их только с --force --delete-emptydir-data, а cluster-autoscaler не освободит узел с ними, пока вы не добавите аннотацию cluster-autoscaler.kubernetes.io/safe-to-evict: "true" через rooms.podAnnotations. Удалённый под комната поднимет снова при следующем запуске: тетради и файлы на месте, переменные Python пропадают. С ReadWriteOnce обслуживание узла приложения — перерыв для всей установки.
    Удаление. helm uninstall убирает объекты чарта, кроме PVC — с persistence.retain: true они остаются, — и не трогает поды и Service, которые создал брокер: они не принадлежат Helm. Удалите их по метке: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
+
    Честные ограничения
    • Одна реплика приложения. База — SQLite, писатель один: отказоустойчивости нет, обновление и отказ узла прерывают работу на время перезапуска пода. Чтобы расти, ставят отдельные установки, по одной на пространство имён, — например, по одной на подразделение.
    • SSO — только через прокси. Своего входа через SSO, LDAP и второго фактора у Colloq нет: его отдают прокси, который подписывает JWT, — см. «Вход через корпоративный SSO». Без такого прокси владелец и преподаватели входят по личным ссылкам; панель можно закрыть на ingress-контроллере — например, oauth2-proxy с корпоративным IdP через аннотации внешней аутентификации ingress-nginx на отдельном Ingress для путей /admin и /api/admin. Участники соревнований и с прокси входят по своему ключу.
    • Ссылка комнаты — общая ссылка. Кто её получил, тот войдёт и по умолчанию сможет запускать код; отозвать её нельзя. Для индивидуально оцениваемых и конфиденциальных работ комнаты не подходят: участники делят ядро, файлы и пользователя Linux.
    • ReadWriteOnce — это один узел. Вся установка помещается на один узел, и его обслуживание — перерыв для всех.
    • У кода студентов нет интернета. pip install в ячейке не работает; библиотеки — только в образах ядер из каталога, которые собираются вне кластера.
    • Нет квоты диска на комнату: том colloq-workspace общий.
    • Сетевые политики — не вся изоляция. Стандартная NetworkPolicy не гарантирует, что под не достучится до служб своего узла, а поды делят ядро Linux узла.
    • Доля карты у GPU-посылки — та, что даёт device plugin. Видеопамять на Kubernetes Colloq не ограничивает и не проверяет: у части MIG своя память, а реплики time-slicing делят память и вычисления карты без всякой границы, и одна посылка может занять память, нужную остальным.
    • Только amd64.
    • Ядра — на Debian 12. Сканер найдёт в них критические находки, которые Debian не исправил; см. «Образы и зеркало реестра».
    • Путь новый. Чарт появился в этом выпуске: проверьте установку в непроизводственном пространстве имён, прежде чем вести в нём занятия.
    
diff --git a/docs/pages/ru/server.html b/docs/pages/ru/server.html
index 97060cd9..a16e1edc 100644
--- a/docs/pages/ru/server.html
+++ b/docs/pages/ru/server.html
@@ -1,7 +1,7 @@
 
    Colloq в вузе ставят двумя способами — выберите тот, что подходит вашей инфраструктуре.
    A / ЕСТЬ КЛАСТЕР KUBERNETES
    В вашем Kubernetes
    Helm-чарт в одно пространство имён: без прав администратора кластера, под Pod Security restricted, с вашим реестром, хранилищем, ingress и секретами из Vault. Для организаций, где кластер ведёт платформенная команда.
    B / ОДНА МАШИНА
    На одной VM
    Готовый образ и colloq-host на выделенной Linux-машине за обратным прокси вуза. Об этом — вся страница ниже.
    
 
    Кратко для ИТ-службы
    Что этоColloq — совместные тетради Jupyter для занятий: веб-приложение и отдельный контейнер с ядром Python для каждой комнаты. Открытый код, лицензия MIT.
    МашинаОдна выделенная VM x86-64 с Ubuntu 22.04 или 24.04 и Docker Engine 28 или новее. Приложение управляет Docker, а это права root на VM, поэтому других служб на ней не держите.
    РесурсыДля начала 16 vCPU, 64 ГБ памяти, 250 ГБ NVMe, без GPU. Этого хватает на группу из 30 человек с личными тетрадями или с соревнованием и на лекцию на 100 человек; расчёт — в разделе «Машина и ресурсы».
    ВходящиеТолько HTTPS от браузеров пользователей к вашему обратному прокси; прокси обращается к приложению по 127.0.0.1:3000. Если пользователи в сети вуза или в VPN, из интернета не нужно ничего.
    ИсходящиеВо время работы основным функциям не нужно ничего: нет ни телеметрии, ни проверки обновлений, ни CDN. Для установки и обновления нужен ghcr.io — или ваше зеркало реестра, или перенос образов файлом. По желанию: модель ИИ (OpenAI или своя внутри сети), импорт тетрадей с GitHub, PyPI или внутреннее зеркало для пакетов соревнований, интернет для кода студентов.
    ДанныеВсё на этой VM в /workspace/colloq: база SQLite, файлы комнат, данные соревнований, резервные копии. Настройки — в /etc/colloq/colloq.env.
    Резервные копииcolloq-host backup вручную или colloq-host backup-timer on — каждую ночь. Уносить копии с машины нужно вашими средствами.
    
 
    Как это устроено
    Браузеры студентов и преподавателейОбратный прокси вуза, TLSКонтейнер Colloq
    Ядро комнаты AЯдро комнаты BПосылка соревнования
    Обратный прокси вуза принимает HTTPS и передаёт запросы приложению. Приложение — один контейнер colloq из образа ghcr.io/colloq-edu/colloq-server: сервер на Node.js, собранный веб-интерфейс и база SQLite. Его порт 3000 опубликован только на 127.0.0.1 этой VM. Запускает и обновляет контейнер colloq-host — небольшой скрипт на хосте, который ставится из того же образа.
    Для каждой комнаты приложение запускает на том же Docker свой контейнер с ядром Jupyter: свои пределы памяти, процессора и числа процессов, смонтирована только папка этой комнаты, uid 1000, без Linux capabilities. Личные тетради студентов считаются во втором контейнере занятия, посылка соревнования — в двух одноразовых контейнерах без сети. Приложение и ядра комнат делят сеть Docker colloq.
    Каталог состояния — /workspace/colloq: data/ (база и ключи), workspace/<комната>/ (файлы комнат), environments/ (окружения, созданные в панели), backups/. Он смонтирован в контейнер по тому же пути. Настройки лежат в /etc/colloq/colloq.env (права 0600), выбранный образ — в /etc/colloq/image. Подключите под /workspace/colloq отдельный диск или раздел: тогда переполнение не заденет систему.
    
-
    Машина и ресурсы
    Отправная точка: 16 vCPU, 64 ГБ памяти, 250 ГБ NVMe, x86-64 (образ собирается только для amd64), Ubuntu 22.04 или 24.04, Docker Engine 28 или новее, GPU не нужен. Часы машины должны синхронизироваться по NTP (без интернета — с внутренним сервером): от времени зависят ссылки входа, cookies и сертификаты. Docker должен сам управлять iptables: сетевой запрет комнат встаёт в его цепочку DOCKER-USER, поэтому не подходят "iptables": false в daemon.json и nftables-бэкенд Docker 29. Память и процессор машины уходят на ядра комнат и посылки; цифры для них ниже выведены из пределов в коде. Само приложение лёгкое, это замерено.
    СценарийvCPUПамятьДиск
    (a) 30 студентов, одна общая тетрадь48 ГБ60 ГБ
    (b) 30 студентов, у каждого личная тетрадь1664 ГБ (не меньше 48 ГБ)100 ГБ
    (c) 30 студентов и соревнование, посылка 2 ядра / 2 ГБ16 (авто даёт 7 слотов)32 ГБ100 ГБ
    (d) три занятия как (c) в один час32 (около 15 слотов)64 ГБ; 160–192 ГБ, если у всех трёх личные тетради200 ГБ
    (e) 100 человек в лекционной комнате4 (8, если студенты запускают ячейки)8 ГБ60 ГБ
    Отправная точка покрывает (a), (b), (c) и (e), а (d) — с более медленной очередью посылок. Предположения: pandas и scikit-learn с настройками по умолчанию, посылка идёт около двух минут.
    Замер приложения (30.09.2026, VM 21 vCPU / 49 ГБ, nginx перед приложением, make load): 500 человек вошли в одну комнату за минуту без единого отказа, вход — 7 мс в середине распределения. Процесс приложения держал 243 МБ памяти и 20 % одного ядра в покое и 50 % ядра, когда двадцать человек печатали одновременно; правка доходила до остальных за 0,17 с (p95). На 300 человек — 198 МБ и те же доли ядра.
    Память считайте так: около 3 ГБ на систему и приложение + пределы комнат (4 ГБ на комнату по умолчанию) + пределы контейнеров личных тетрадей + слоты × (память посылки + 0,25 ГБ) + 1,6 ГБ на время подготовки пакетов. Docker ничего не резервирует, поэтому Colloq сам не запустит ядро, если его предел вместе с пределами уже работающих контейнеров не помещается в память машины за вычетом 1 ГБ. Процессор делится, а не резервируется: у комнаты потолок в 2 ядра.
    • Личные тетради: около 1–1,5 ГБ и 0,25 ядра на студента. Все личные тетради занятия живут в одном контейнере и по умолчанию получают на всех столько же, сколько комната, — 4 ГБ. Для группы поднимите «Память на класс» в панели: «Ресурсы» → «Личные тетради студентов» (или в правилах конкретного занятия).
    • Соревнования: сколько посылок идёт одновременно («Исполнителей одновременно» во вкладке «Ресурсы»), по умолчанию считается само: меньшее из (ядра − 2) / ядра посылки и (память − 2 ГБ) / (память посылки + 0,25 ГБ), но не больше 32. 30 посылок по две минуты проходят примерно за 9 минут на 7 слотах и примерно за 20 минут на 3 слотах (машина 8 vCPU / 16 ГБ).
    • Комната держит память, пока в ней кто-то есть, и ещё 2 часа после того, как все ушли: конец пары ядро не останавливает. Поэтому занятия подряд накладываются друг на друга.
    • Диск: образ kaggle-base занимает около 1 ГБ, base-gpu — около 9 ГБ; дальше растут файлы комнат, посылки соревнований и резервные копии. Страницы занятий держат свои копии опубликованных тетрадей и файлов в data/page-files/: около 0,3–0,8 ГБ на курс из 33 занятий, одинаковый файл в разных страницах хранится один раз. Квоты на комнату нет, см. наблюдение.
    • GPU необязателен. Он нужен только окружениям с GPU (base-gpu): по одной карте, 16 ГБ памяти и около 10 ГБ диска на каждую одновременно работающую GPU-комнату, плюс драйвер NVIDIA и NVIDIA Container Toolkit (colloq-host ставит его сам через apt). Личные тетради и посылки соревнований GPU не получают никогда.
    Во вкладке «Ресурсы» есть шкала памяти: работающие комнаты, личные тетради и слоты соревнований против памяти машины за вычетом 1 ГБ; если памяти не хватает, шкала так и пишет. Там же показан свободный диск раздела с данными, с предупреждением, когда свободно меньше 15 % или 10 ГБ; GPU шкала не показывает.
    
+
    Машина и ресурсы
    Отправная точка: 16 vCPU, 64 ГБ памяти, 250 ГБ NVMe, x86-64 (образ собирается только для amd64), Ubuntu 22.04 или 24.04, Docker Engine 28 или новее, GPU не нужен. Часы машины должны синхронизироваться по NTP (без интернета — с внутренним сервером): от времени зависят ссылки входа, cookies и сертификаты. Docker должен сам управлять iptables: сетевой запрет комнат встаёт в его цепочку DOCKER-USER, поэтому не подходят "iptables": false в daemon.json и nftables-бэкенд Docker 29. Память и процессор машины уходят на ядра комнат и посылки; цифры для них ниже выведены из пределов в коде. Само приложение лёгкое, это замерено.
    СценарийvCPUПамятьДиск
    (a) 30 студентов, одна общая тетрадь48 ГБ60 ГБ
    (b) 30 студентов, у каждого личная тетрадь1664 ГБ (не меньше 48 ГБ)100 ГБ
    (c) 30 студентов и соревнование, посылка 2 ядра / 2 ГБ16 (авто даёт 7 слотов)32 ГБ100 ГБ
    (d) три занятия как (c) в один час32 (около 15 слотов)64 ГБ; 160–192 ГБ, если у всех трёх личные тетради200 ГБ
    (e) 100 человек в лекционной комнате4 (8, если студенты запускают ячейки)8 ГБ60 ГБ
    Отправная точка покрывает (a), (b), (c) и (e), а (d) — с более медленной очередью посылок. Предположения: pandas и scikit-learn с настройками по умолчанию, посылка идёт около двух минут.
    Замер приложения (30.09.2026, VM 21 vCPU / 49 ГБ, nginx перед приложением, make load): 500 человек вошли в одну комнату за минуту без единого отказа, вход — 7 мс в середине распределения. Процесс приложения держал 243 МБ памяти и 20 % одного ядра в покое и 50 % ядра, когда двадцать человек печатали одновременно; правка доходила до остальных за 0,17 с (p95). На 300 человек — 198 МБ и те же доли ядра.
    Память считайте так: около 3 ГБ на систему и приложение + пределы комнат (4 ГБ на комнату по умолчанию) + пределы контейнеров личных тетрадей + слоты × (память посылки + 0,25 ГБ) + 1,6 ГБ на время подготовки пакетов. Docker ничего не резервирует, поэтому Colloq сам не запустит ядро, если его предел вместе с пределами уже работающих контейнеров не помещается в память машины за вычетом 1 ГБ. Процессор делится, а не резервируется: у комнаты потолок в 2 ядра.
    • Личные тетради: около 1–1,5 ГБ и 0,25 ядра на студента. Все личные тетради занятия живут в одном контейнере и по умолчанию получают на всех столько же, сколько комната, — 4 ГБ. Для группы поднимите «Память на класс» в панели: «Ресурсы» → «Личные тетради студентов» (или в правилах конкретного занятия).
    • Соревнования: сколько посылок идёт одновременно («Исполнителей одновременно» во вкладке «Ресурсы»), по умолчанию считается само: меньшее из (ядра − 2) / ядра посылки и (память − 2 ГБ) / (память посылки + 0,25 ГБ), но не больше 32. 30 посылок по две минуты проходят примерно за 9 минут на 7 слотах и примерно за 20 минут на 3 слотах (машина 8 vCPU / 16 ГБ).
    • Комната держит память, пока в ней кто-то есть, и ещё 2 часа после того, как все ушли: конец пары ядро не останавливает. Поэтому занятия подряд накладываются друг на друга.
    • Диск: образ kaggle-base занимает около 1 ГБ, base-gpu — около 9,9 ГБ, kaggle-gpu — ещё около 2,8 ГБ поверх base-gpu, всего около 12,7 ГБ; дальше растут файлы комнат, посылки соревнований и резервные копии. Страницы занятий держат свои копии опубликованных тетрадей и файлов в data/page-files/: около 0,3–0,8 ГБ на курс из 33 занятий, одинаковый файл в разных страницах хранится один раз. Квоты на комнату нет, см. наблюдение.
    • GPU необязателен. Он нужен только окружениям с GPU (base-gpu, kaggle-gpu): по одной карте и 16 ГБ памяти на каждую одновременно работающую GPU-комнату, плюс драйвер NVIDIA версии 580 или новее и NVIDIA Container Toolkit (colloq-host сам ставит через apt только его, не драйвер). В обоих окружениях torch 2.14.1 собран под CUDA 13.0: на старом драйвере PyTorch импортируется, но карты не видит, а версию драйвера nvidia-smi показывает в шапке. Личные тетради GPU не получают никогда. Посылки соревнований получают долю карты, только если владелец отдал посылкам карты во вкладке «Ресурсы», а в соревновании включён переключатель «Видеокарта»; отданная посылкам карта комнатам больше не служит. См. видеокарту для посылок.
    Во вкладке «Ресурсы» есть шкала памяти: работающие комнаты, личные тетради и слоты соревнований против памяти машины за вычетом 1 ГБ; если памяти не хватает, шкала так и пишет. Там же показан свободный диск раздела с данными, с предупреждением, когда свободно меньше 15 % или 10 ГБ. Видеопамять шкала не считает; карты с их памятью и тем, кому каждая служит, перечислены в блоке «Видеокарты для посылок» той же вкладки. Названия и память карт читаются при запуске сервера и затем раз в десять минут, через одноразовый контейнер образа ядра, которому даны все карты: в контейнере самого сервера nvidia-smi нет; пока на машине нет ни одного образа ядра, а у экземпляра MIG всегда, размер карты остаётся неизвестным.
    
 
    Входящие и исходящие соединения
    Входящие соединения:
    Откуда и кудаПортЗачем
    Браузеры пользователей → обратный прокси443/TCP; 80 — только для перенаправления на HTTPSВсё приложение: страницы, API, WebSocket, server-sent events.
    Обратный прокси → приложение127.0.0.1:3000, если прокси на этой же VM; иначе внутренний адрес VM, порт 3000 (COLLOQ_BIND)HTTP без TLS внутри машины или доверенного сегмента.
    Администраторы → VM22/TCPSSH и colloq-host; только из сети ИТ-службы.
    Интернет → VM—Не нужен, если пользователи в сети вуза или в VPN. Порты 80/443 из интернета нужны, только если прокси сам получает сертификат Let's Encrypt.
    Jupyter ядер и остальные служебные порты не публикуются: они живут внутри сети Docker. Сторонних обращений нет: шрифты, просмотрщик PDF и plotly приложение отдаёт само, и браузеры обращаются только к адресу Colloq.
    Исходящие соединения:
    КудаЗачем и когдаНужноМожно ли заменить
    ghcr.io, pkg-containers.githubusercontent.comОбраз приложения colloq-server и образы ядер colloq-kernel: установка и обновлениеДа, если не переносить образы файломДа: зеркало реестра в COLLOQ_IMAGE и KERNEL_IMAGE_REPO; без сети — перенос файлом
    get.docker.com, download.docker.comУстановка Docker, если его нет на машинеНет, если Docker уже стоитСтавьте Docker из своего репозитория
    Docker Hub (registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com), deb.debian.org, pypi.org, files.pythonhosted.orgСборка образа ядра на месте: окружение из панели или опубликованный образ, который не удалось скачатьНетЧастично: строки --index-url в списке пакетов окружения; готовые образы ядер из реестра
    nvidia.github.ioNVIDIA Container Toolkit при первой установке на машине с GPUТолько для GPUПоставьте пакет заранее
    Модель ИИ, по умолчанию api.openai.comОракул — когда его спрашиваютНетДа: OPENAI_BASE_URL или панель — Ollama, vLLM или другой совместимый с OpenAI адрес внутри сети
    api.github.com, raw.githubusercontent.comИмпорт тетради с GitHub в панели — по кнопкеНетНет, только github.com
    pypi.org, files.pythonhosted.org«Свои пакеты» участников соревнований — при подготовке набораНетДа: DEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS
    Интернет из ядер комнатpip install, датасеты и API в коде студентов — во время занятияНетМожно отключить: COLLOQ_ROOM_NETWORK=none, см. ниже
    
 
    Сеть для кода студентов
    • Посылки соревнований исполняются вовсе без сети.
    • Комнаты и личные тетради по умолчанию выходят в интернет (pip install, датасеты, API), а частные адреса для них закрыты: 10/8, 172.16/12, 192.168/16, 100.64/10, 169.254/16 с метаданными облака, 127/8, multicast, сама VM и соседние комнаты. Соединение туда сразу получает отказ; открытым для любого адреса остаётся только DNS, порт 53. Правила ставит само приложение — в собственные цепочки iptables COLLOQ-ROOMS-*, в которые ведут DOCKER-USER и INPUT. Если поставить их не удалось, комнаты не запускаются.
    • Публичные диапазоны вуза. Код студентов выходит в сеть с адреса VM, и службы, которые доверяют адресам вуза (подписки библиотеки, внутренние сайты на публичных адресах), примут его за своего. Перечислите такие диапазоны: KERNEL_BLOCKED_CIDRS=192.0.2.0/24,198.51.100.0/24.
    • COLLOQ_ROOM_NETWORK=none — у ядер комнат и личных тетрадей совсем нет исходящей сети, до них доходит только сервер Colloq. pip install в ячейке тогда не работает: нужные библиотеки заранее кладите в окружение (панель → «Окружения»).
    • COLLOQ_ROOM_NETWORK=open снимает запрет частных адресов — только для доверенных групп, которым нужна локальная сеть.
    
 
    Исходящий прокси, внутренний CA и зеркало PyPI
    Если наружу можно выйти только через прокси, задайте HTTPS_PROXY, HTTP_PROXY и NO_PROXY (понимаются и в нижнем регистре). Через прокси идут все исходящие HTTP-запросы сервера — к модели ИИ и к GitHub, — и те же значения получает pip при подготовке пакетов. Частные адреса и внутренние имена Colloq (ядра, контейнеры) идут мимо прокси сами; в NO_PROXY перечислите внутренние домены, например своей модели и зеркала PyPI. NODE_EXTRA_CA_CERTS — путь к файлу PEM с сертификатами внутреннего CA на хосте: colloq-host монтирует его в контейнер по тому же пути, и его тоже получает pip. Тот же файл положите и в системное хранилище: образы скачивает сам Docker, и ему нужны и этот CA, и свой прокси.
    cp campus-ca.pem /usr/local/share/ca-certificates/campus-ca.crt
@@ -96,7 +96,7 @@ 
    Установка
    Всё ниже выполняетс
 colloq-host link
 colloq-host status
 colloq-host backup-timer on
    doctor ничего не меняет: по строке PASS, WARN или FAIL на каждую проверку — архитектура, версия Docker и его сетевой экран, память и диск, порт 3000, DNS, синхронизация часов, доступ к ghcr.io, PUBLIC_URL и доверенные прокси. Исправьте каждую строку FAIL до первого занятия. up ставит недостающее, скачивает образ, записывает настройки в /etc/colloq/colloq.env и запускает контейнер colloq. Образ ядра по умолчанию (base) скачивается из ghcr.io/colloq-edu/colloq-kernel, а если не вышло — собирается на месте; это несколько минут в фоне, и панель тем временем уже работает. Если прокси на другой машине, добавьте к up COLLOQ_BIND и TRUSTED_PROXIES, см. «Адреса клиентов».
    colloq-host link печатает адрес и, пока у сервера нет владельца, ссылку установки …/admin/t/…. Это ключ от сервера: не отправляйте его в общий чат. Откройте ссылку, введите имя и почту — и вы владелец. Преподавателей добавьте в панели, в разделе «Преподаватели»: каждый получит личную ссылку входа.
    colloq-host status показывает /api/health, образы ядер и контейнеры комнат; сервер готов к занятию, когда в ответе "ok":true. Перед первой группой создайте две комнаты, откройте их с двух устройств из сети студентов, запустите ячейку, загрузите файл, а затем проверьте резервную копию и восстановление на запасной VM.
    
-
    Настройки
    colloq-host up записывает в /etc/colloq/colloq.env (права 0600) переменные из своего окружения: непустое значение заменяет прежнее, а отсутствующее ничего не стирает. Там же он запоминает и собственные настройки — COLLOQ_HOME, COLLOQ_BIND, — чтобы следующий update действовал так же. Чтобы убрать строку, отредактируйте файл. Изменения применяет повторный colloq-host up: контейнер приложения пересоздаётся, только если изменились настройки или образ, а ядра комнат при этом продолжают работать. Оракул, ресурсы комнат и слоты соревнований владелец меняет и в панели, без перезапуска; сохранённое в панели главнее файла.
    ПеременнаяЧто задаётПо умолчанию
    Адрес и прокси
    COLLOQ_TUNNELnone — адрес даёт ваш прокси. Остальные режимы (relay, cloudflare, direct) — для аренды и публикации без своего прокси.auto; вне Vast без PUBLIC_URL — none с предупреждением
    PUBLIC_URLВнешний адрес, из которого строятся ссылки.—
    TRUSTED_PROXIESАдреса прокси, чей X-Forwarded-For принимается: IP, подсети, loopback, private, none.loopback; при порте на loopback colloq-host добавляет шлюз сети colloq
    SHARED_ADDRESSESПодсети NAT, для которых не действуют пределы на адрес.пусто
    HSTS0 — не отправлять Strict-Transport-Security.включено
    TRUST_CF_CONNECTING_IP1 — принимать CF-Connecting-IP; только для туннелей Cloudflare самого Colloq, не за вашим прокси.выключено
    COLLOQ_BINDГде хост публикует порт 3000; нужен, если прокси на другой машине. Читает только colloq-host.127.0.0.1
    Исходящие соединения
    HTTPS_PROXY, HTTP_PROXY, NO_PROXYИсходящий прокси для модели ИИ, импорта с GitHub и pip.—
    NODE_EXTRA_CA_CERTSФайл PEM внутреннего CA (путь внутри контейнера).—
    DEPENDENCY_INDEX_URLИндекс для «своих пакетов» соревнований.https://pypi.org/simple
    DEPENDENCY_FILES_HOSTSХосты, которые отдают файлы пакетов, через запятую.—
    COLLOQ_ROOM_NETWORKnone — у комнат нет сети; open — нет запрета частных адресов.пусто: интернет есть, частные адреса закрыты
    KERNEL_BLOCKED_CIDRSДополнительные подсети, закрытые для комнат.—
    COLLOQ_HELPER_IMAGEОбраз служебного контейнера, который ставит правила iptables.образ самого сервера
    COLLOQ_OFFLINE1 — ничего не скачивать, образы приходят через colloq-host load; его и записывает load.выключено
    Образы и резервные копии
    COLLOQ_HOMEКаталог состояния; задаётся до первого up. Читает только colloq-host./workspace/colloq
    COLLOQ_IMAGEОбраз первого up; дальше образ выбирает colloq-host update (/etc/colloq/image).—
    KERNEL_IMAGE_REPOОткуда брать готовые образы ядер: <repo>:v<версия>-<окружение>; none — собирать все окружения на месте.ghcr.io/colloq-edu/colloq-kernel
    COLLOQ_REGISTRY_USER, COLLOQ_REGISTRY_TOKENВход в закрытый реестр (токен только на чтение); читает только colloq-host.—
    BACKUP_KEEPСколько последних копий хранить; 0 — все.14
    BACKUP_AGE_RECIPIENTОткрытый ключ age: копии шифруются для него. Восстановление — с BACKUP_AGE_IDENTITY, путём к закрытому ключу.—
    COLLOQ_ALLOW_SCHEMA_DOWNGRADE1 — открыть базу, которую записала более новая версия. Без него старый образ на такой базе не запускается; обычный откат — restore --image с копией до обновления.—
    COLLOQ_BACKUP_ATВремя ежедневной копии; читает backup-timer on.03:30
    COLLOQ_BACKUP_BEFORE_UPDATE0 — не делать копию перед update; читает update.1
    COLLOQ_FREEZE_UPDATES1 — отключить автообновления системы и закрепить пакеты NVIDIA. Его ставит только on-start для Vast; на сервере вуза не задавайте.выключено
    Занятия
    INSTITUTIONНазвание рядом с логотипом.—
    UI_LANGUAGEЯзык интерфейса (ru, en), пока владелец не выберет его в панели.ru
    TZЧасовой пояс: границы дня для дневных норм, даты.Europe/Moscow
    ADMIN_EMAILПочта, которую форма владельца предлагает заранее.—
    MAX_UPLOAD_MBПредел одного файла, загружаемого в комнату.50
    MAX_SESSION_MBПредел всех загрузок в одну комнату; это не квота диска.1024
    KERNEL_MEMПамять ядра комнаты; KERNEL_MEM_<ОКРУЖЕНИЕ> — для одного окружения.4g, у GPU-окружений 16g
    KERNEL_CPUSЯдра процессора на комнату.2
    KERNEL_OWN_MAX, KERNEL_OWN_PIDS, KERNEL_OWN_IDLE_MINКонтейнер личных тетрадей: сколько в нём ядер, потолок процессов, минуты простоя до остановки ядра.60, 2048, 30
    KERNEL_ROOM_IDLE_MINСколько минут пустое занятие (никого нет, ничего не считается) держит свои ядра; 0 — держать. Меняется и во вкладке «Ресурсы».120
    KERNEL_ENV, KERNEL_PRELOADОкружение по умолчанию; какие окружения подготовить при старте.base; как KERNEL_ENV
    KERNEL_GPUSКарты для GPU-комнат.все найденные
    OPEN_SEMINAR_CREATIONРазрешить создавать комнаты через API без входа преподавателя.false
    SESSION_SECRETКлюч подписи ссылок; новое значение делает недействительными все выданные ссылки.создаётся сам, data/session-secret
    Оракул
    AI_PROVIDERopenai, ollama, vllm, openrouter или custom.openai
    OPENAI_API_KEYКлюч модели; без него оракул выключен (Ollama и vLLM работают без ключа).—
    OPENAI_BASE_URLАдрес модели.https://api.openai.com/v1
    OPENAI_MODELМодель.gpt-4o-mini
    Вход через прокси (SSO)
    AUTH_JWT_JWKS_URL, AUTH_JWT_AUDIENCE, AUTH_JWT_ISSUER и другие AUTH_JWT_*Вход через Teleport или другой прокси, который подписывает JWT: преподаватели узнаются по почте, студенты входят под своим именем. Как это устроено и какие поля токена читаются — в руководстве для Kubernetes; на машине те же переменные передаются colloq-host up. Если приложение публикует Teleport на этой же машине: COLLOQ_TUNNEL=none, а PUBLIC_URL — адрес приложения в Teleport.выключено
    
+
    Настройки
    colloq-host up записывает в /etc/colloq/colloq.env (права 0600) переменные из своего окружения: непустое значение заменяет прежнее, а отсутствующее ничего не стирает. Там же он запоминает и собственные настройки — COLLOQ_HOME, COLLOQ_BIND, — чтобы следующий update действовал так же. Чтобы убрать строку, отредактируйте файл. Изменения применяет повторный colloq-host up: контейнер приложения пересоздаётся, только если изменились настройки или образ, а ядра комнат при этом продолжают работать. Оракул, ресурсы комнат и слоты соревнований владелец меняет и в панели, без перезапуска; сохранённое в панели главнее файла.
    ПеременнаяЧто задаётПо умолчанию
    Адрес и прокси
    COLLOQ_TUNNELnone — адрес даёт ваш прокси. Остальные режимы (relay, cloudflare, direct) — для аренды и публикации без своего прокси.auto; вне Vast без PUBLIC_URL — none с предупреждением
    PUBLIC_URLВнешний адрес, из которого строятся ссылки.—
    TRUSTED_PROXIESАдреса прокси, чей X-Forwarded-For принимается: IP, подсети, loopback, private, none.loopback; при порте на loopback colloq-host добавляет шлюз сети colloq
    SHARED_ADDRESSESПодсети NAT, для которых не действуют пределы на адрес.пусто
    HSTS0 — не отправлять Strict-Transport-Security.включено
    TRUST_CF_CONNECTING_IP1 — принимать CF-Connecting-IP; только для туннелей Cloudflare самого Colloq, не за вашим прокси.выключено
    COLLOQ_BINDГде хост публикует порт 3000; нужен, если прокси на другой машине. Читает только colloq-host.127.0.0.1
    Исходящие соединения
    HTTPS_PROXY, HTTP_PROXY, NO_PROXYИсходящий прокси для модели ИИ, импорта с GitHub и pip.—
    NODE_EXTRA_CA_CERTSФайл PEM внутреннего CA (путь внутри контейнера).—
    DEPENDENCY_INDEX_URLИндекс для «своих пакетов» соревнований.https://pypi.org/simple
    DEPENDENCY_FILES_HOSTSХосты, которые отдают файлы пакетов, через запятую.—
    COLLOQ_ROOM_NETWORKnone — у комнат нет сети; open — нет запрета частных адресов.пусто: интернет есть, частные адреса закрыты
    KERNEL_BLOCKED_CIDRSДополнительные подсети, закрытые для комнат.—
    COLLOQ_HELPER_IMAGEОбраз служебного контейнера, который ставит правила iptables.образ самого сервера
    COLLOQ_OFFLINE1 — ничего не скачивать, образы приходят через colloq-host load; его и записывает load.выключено
    Образы и резервные копии
    COLLOQ_HOMEКаталог состояния; задаётся до первого up. Читает только colloq-host./workspace/colloq
    COLLOQ_IMAGEОбраз первого up; дальше образ выбирает colloq-host update (/etc/colloq/image).—
    KERNEL_IMAGE_REPOОткуда брать готовые образы ядер: <repo>:v<версия>-<окружение>; none — собирать все окружения на месте.ghcr.io/colloq-edu/colloq-kernel
    COLLOQ_REGISTRY_USER, COLLOQ_REGISTRY_TOKENВход в закрытый реестр (токен только на чтение); читает только colloq-host.—
    BACKUP_KEEPСколько последних копий хранить; 0 — все.14
    BACKUP_AGE_RECIPIENTОткрытый ключ age: копии шифруются для него. Восстановление — с BACKUP_AGE_IDENTITY, путём к закрытому ключу.—
    COLLOQ_ALLOW_SCHEMA_DOWNGRADE1 — открыть базу, которую записала более новая версия. Без него старый образ на такой базе не запускается; обычный откат — restore --image с копией до обновления.—
    COLLOQ_BACKUP_ATВремя ежедневной копии; читает backup-timer on.03:30
    COLLOQ_BACKUP_BEFORE_UPDATE0 — не делать копию перед update; читает update.1
    COLLOQ_FREEZE_UPDATES1 — отключить автообновления системы и закрепить пакеты NVIDIA. Его ставит только on-start для Vast; на сервере вуза не задавайте.выключено
    Занятия
    INSTITUTIONНазвание рядом с логотипом.—
    UI_LANGUAGEЯзык интерфейса (ru, en), пока владелец не выберет его в панели.ru
    TZЧасовой пояс: границы дня для дневных норм, даты.Europe/Moscow
    ADMIN_EMAILПочта, которую форма владельца предлагает заранее.—
    MAX_UPLOAD_MBПредел одного файла, загружаемого в комнату.50
    MAX_SESSION_MBПредел всех загрузок в одну комнату; это не квота диска.1024
    KERNEL_MEMПамять ядра комнаты; KERNEL_MEM_<ОКРУЖЕНИЕ> — для одного окружения.4g, у GPU-окружений 16g
    KERNEL_CPUSЯдра процессора на комнату.2
    KERNEL_OWN_MAX, KERNEL_OWN_PIDS, KERNEL_OWN_IDLE_MINКонтейнер личных тетрадей: сколько в нём ядер, потолок процессов, минуты простоя до остановки ядра.60, 2048, 30
    KERNEL_ROOM_IDLE_MINСколько минут пустое занятие (никого нет, ничего не считается) держит свои ядра; 0 — держать. Меняется и во вкладке «Ресурсы».120
    KERNEL_ENV, KERNEL_PRELOADОкружение по умолчанию; какие окружения подготовить при старте.base; как KERNEL_ENV
    KERNEL_GPUSКарты для GPU-комнат и, из них, отданные посылкам соревнований.все найденные
    COMPETITION_SLOTSСколько посылок исполняется одновременно: число от 1 до 64 или auto — посчитать по машине. Меняется и во вкладке «Ресурсы».auto
    COMPETITION_GPUS, COMPETITION_GPU_PER_DEVICEКарты, отданные посылкам соревнований (элементы KERNEL_GPUS через запятую), и сколько посылок делят одну карту одновременно, от 1 до 8; изменение касается прогонов, взятых после него, а посылка, уже идущая на карте, сохраняет свою долю. Меняются и во вкладке «Ресурсы», см. видеокарту для посылок.нет; 1
    OPEN_SEMINAR_CREATIONРазрешить создавать комнаты через API без входа преподавателя.false
    SESSION_SECRETКлюч подписи ссылок; новое значение делает недействительными все выданные ссылки.создаётся сам, data/session-secret
    Оракул
    AI_PROVIDERopenai, ollama, vllm, openrouter или custom.openai
    OPENAI_API_KEYКлюч модели; без него оракул выключен (Ollama и vLLM работают без ключа).—
    OPENAI_BASE_URLАдрес модели.https://api.openai.com/v1
    OPENAI_MODELМодель.gpt-4o-mini
    Вход через прокси (SSO)
    AUTH_JWT_JWKS_URL, AUTH_JWT_AUDIENCE, AUTH_JWT_ISSUER и другие AUTH_JWT_*Вход через Teleport или другой прокси, который подписывает JWT: преподаватели узнаются по почте, студенты входят под своим именем. Как это устроено и какие поля токена читаются — в руководстве для Kubernetes; на машине те же переменные передаются colloq-host up. Если приложение публикует Teleport на этой же машине: COLLOQ_TUNNEL=none, а PUBLIC_URL — адрес приложения в Teleport.выключено
    
 
    Резервные копии
    colloq-host backup                # копия сейчас
 colloq-host backup-timer on       # каждый день в 03:30; другое время — COLLOQ_BACKUP_AT=02:00
 colloq-host backup-timer status
    Таймер — это служба systemd: копия делается по часовому поясу машины, а пропущенная, пока машина была выключена, — сразу после включения. Копия — это пара файлов в /workspace/colloq/backups/: согласованный снимок базы colloq-<время>.db и архив colloq-<время>-files.tar.gz с файлами комнат, картинками вывода, файлами страниц занятий, данными и посылками соревнований, наборами пакетов, ключом подписи ссылок, setup-token и списками пакетов своих окружений. Файлы копируются на ходу и у идущего занятия могут измениться во время копирования, поэтому таймер лучше ставить на ночь. Хранятся 14 последних копий (BACKUP_KEEP), более старые удаляются после успешной копии. Образов ядер в копии нет: их скачают или соберут заново.
    Копия — это ключи от сервера: в ней ключ подписи, setup-token и база с ключами входа преподавателей и ключом ИИ, если его вводили в панели. Файлы создаются с правами 0600. Уносите их с машины своими средствами (rsync, restic или borg в хранилище вуза) и храните зашифрованными: диск VM — не резервная копия. Зашифровать копию может и сам Colloq: задайте BACKUP_AGE_RECIPIENT — открытый ключ age, и обе части копии запишутся как .age. Закрытый ключ держите вне машины; для восстановления укажите путь к нему: BACKUP_AGE_IDENTITY=/root/colloq-backup.key REPLACE=1 colloq-host restore ….
    Восстановление: положите пару файлов в /workspace/colloq/backups/ и выполните команду ниже. colloq-host остановит Colloq, восстановит базу и файлы и запустит его снова; REPLACE=1 нужен, если на машине уже есть база. Проверяйте восстановление на запасной VM хотя бы раз в семестр.
    REPLACE=1 colloq-host restore backups/colloq-20260930-033000.db
    
diff --git a/docs/pages/ru/troubleshooting.html b/docs/pages/ru/troubleshooting.html
index 81859c97..e04c8331 100644
--- a/docs/pages/ru/troubleshooting.html
+++ b/docs/pages/ru/troubleshooting.html
@@ -6,7 +6,7 @@ 
    Тетрадь открылась, а Python не запуска
 sudo k3s kubectl -n colloq get events --sort-by=.lastTimestamp
    Проверьте, работает ли брокер, доступен ли закреплённый image digest, хватает ли CPU/памяти/GPU и корректен ли mount workspace. ImagePullBackOff обычно требует проверки registry и прав, Pending — причин планирования в событиях. Память комнаты резервируется целиком, поэтому Insufficient memory в событиях значит, что узлу не хватает памяти, уже обещанной комнатам: уменьшите память этой или соседних комнат в разделе «Ресурсы» (см. ресурсы комнаты). Такой Pod брокер ждёт 20 секунд, затем удаляет его и сообщает причину в комнату: преподаватель видит, сколько памяти, ядер или какой видеокарты не нашлось, студенты — короткое сообщение без подробностей. В журнале приложения это строка did not start … [unschedulable: memory] (или cpu, gpu, other). Health брокера проверяет доступ к API, а не весь путь исполнения.
    Не заменяйте недоступную ревизию другой без осознанного решения и не подключайте комнаты к общему Jupyter. После устранения причины выполните sudo scripts/cluster.sh smoke и настоящую ячейку через приложение.
    На сервере с готовым образом начните с colloq-host status и colloq-host logs, см. «Наблюдение, журналы и перезагрузка».
    На локальном занятии (pip install colloq) начните с colloq doctor: он проверяет программы, Docker, порт, образ ядра, файлы и место на диске. Журнал занятия показывает colloq logs, контейнеры комнат — docker ps --filter name=colloq-room-. Если комната пишет, что не удалось закрыть ей доступ к локальным адресам, Docker не пустил короткий привилегированный контейнер, который ставит этот запрет; причину называет само сообщение, а про запрет и выход COLLOQ_ROOM_NETWORK=open рассказывает раздел «Занятие на своём компьютере».
    
 
    NameError или изменившиеся данные
    Возможно, не выполнена подготовительная ячейка, ядро перезапустилось или другой участник изменил общие значения. Проверьте порядок запуска. В Консилиуме каждая попытка работает с личными копиями данных, а между попытками сохраняются файлы, настройки модулей и то, что скопировать не вышло, — о последнем попытка читает строкой в своём выводе.
    
 
    Не устанавливается пакет или не скачиваются данные
    В production исходящая сеть ядра по умолчанию ограничена DNS, а root filesystem защищён от записи. Подготовьте библиотеку в образе окружения, а данные загрузите в комнату или используйте одобренный оператором outbound-профиль. Не отключайте сетевую защиту всего namespace ради одного импорта.
    На локальном занятии интернет комнатам открыт, а локальные адреса — нет: зеркало пакетов, база данных или сервер с данными в вашей сети недоступны, и соединение сразу получает отказ. Положите данные в комнату или, если занятие доверенное, допишите в .env строку COLLOQ_ROOM_NETWORK=open и перезапустите (см. «Занятие на своём компьютере»).
    
-
    CUDA недоступна
    Проверьте драйвер гостевой VM, NVIDIA RuntimeClass, allocatable devices, наличие свободного устройства и GPU-метаданные каталога. CPU-окружение само не становится GPU-окружением.
    nvidia-smi
+
    CUDA недоступна
    Проверьте драйвер гостевой VM, NVIDIA RuntimeClass, allocatable devices, наличие свободного устройства и GPU-метаданные каталога. CPU-окружение само не становится GPU-окружением. Поставляемые GPU-окружения несут torch 2.14.1, собранный под CUDA 13.0, которому нужен драйвер 580 или новее: версию драйвера и его CUDA nvidia-smi пишет в шапке, а на старом драйвере PyTorch импортируется, но torch.cuda.is_available() возвращает False.
    nvidia-smi
 sudo k3s kubectl get runtimeclass
 sudo k3s kubectl describe node
 sudo scripts/cluster.sh gpu-preflight
    Если GPU занят комнатой, preflight не выселяет её. Повторите проверку в согласованное время со свободной картой. Реальное выполнение CUDA необходимо даже при корректном nvidia-smi.
    
diff --git a/docs/pages/ru/vast.html b/docs/pages/ru/vast.html
index 49be86dc..e74ce0d6 100644
--- a/docs/pages/ru/vast.html
+++ b/docs/pages/ru/vast.html
@@ -1,4 +1,4 @@
-
    Готовый образ: make vast-vm
    Быстрый путь — make vast-vm HOST=class1 из checkout репозитория, с VAST_TOKEN и SSH-ключом в .env. Команда арендует Vast VM через API, запускает на ней готовый образ colloq-vast (тот же образ, что colloq-server) под управлением colloq-host и печатает адрес занятия и ссылку владельца. Каждая комната получает свой контейнер ядра на Docker этой VM. Адрес занятию обычно даёт ваш relay; настройки — токен relay, ключ ИИ, COLLOQ_IMAGE — заполните один раз в backups/vast-onstart.sh. На самой VM обслуживание — colloq-host status, backup, update — такое же, как на сервере в вузе; подробности аренды — в deploy/vast/README.md. Данные живут на диске VM и пропадают вместе с арендой: уносите копии до её удаления.
    
+
    Готовый образ: make vast-vm
    Быстрый путь — make vast-vm HOST=class1 из checkout репозитория, с VAST_TOKEN и SSH-ключом в .env. Команда арендует Vast VM через API, запускает на ней готовый образ colloq-vast (тот же образ, что colloq-server) под управлением colloq-host и печатает адрес занятия и ссылку владельца. Каждая комната получает свой контейнер ядра на Docker этой VM. Адрес занятию обычно даёт ваш relay; настройки — токен relay, ключ ИИ, COLLOQ_IMAGE — заполните один раз в backups/vast-onstart.sh. На самой VM обслуживание — colloq-host status, backup, update — такое же, как на сервере в вузе; подробности аренды — в deploy/vast/README.md. Данные живут на диске VM и пропадают вместе с арендой: уносите копии до её удаления.
    Для GPU-комнат или посылок на GPU арендуйте карты и диск, которые нужны образам CUDA: make vast-vm HOST=class1 GPUS=2 GPU_RAM=24 DISK=100 просит не меньше двух карт по 24 ГБ и 100 ГБ диска, потому что base-gpu (около 9,9 ГБ) и kaggle-gpu (ещё около 2,8 ГБ) собираются прямо на VM. Для любого занятия make vast-vm берёт только предложения с Max CUDA 13.0 и выше: torch 2.14.1 из GPU-окружений собран под CUDA 13.0 и требует драйвер NVIDIA 580 или новее. DISK= и MAX_PRICE= заменяют значения из .env для одной аренды, а OFFER= арендует предложение с этим номером, где бы на рынке оно ни стояло. В backups/vast-onstart.sh задайте KERNEL_PRELOAD=base,base-gpu,kaggle-gpu, а когда занятие поднимется, отдайте посылкам карты в панели, см. видеокарту для посылок.
    
 
    Подготовка до аренды
    Путь k3s использует полную Vast VM с systemd, а не обычный Docker instance. VM содержит один узел k3s. Проверьте доступность VM-предложений, рабочий NVIDIA-драйвер, зарегистрированный SSH-ключ, цену и дисковое пространство.
    На доверенной операторской машине нужны манифест выпуска и клон публичного репозитория github.com/colloq-edu/colloq, в котором есть его точный sourceCommit. vast-up упаковывает инструменты именно из этого коммита. Приложение не получает ключ аккаунта Vast.
    В конфигурации подготовьте VAST_TOKEN, SSH-ключ и параметры существующего relay. Для приватных образов задайте VAST_REGISTRY_CONFIG с отдельными pull-only правами и VAST_REGISTRY_READ_ONLY=1. Файл должен содержать только Docker auths, без credential helpers. Синтаксическая проверка не доказывает действительность прав — загрузку проверяет preflight.
    
 
    Запуск и наблюдение
    Для выпуска, чьи образы действительно доступны анонимно:
    VAST_PUBLIC_IMAGES=1 RELEASE=/absolute/path/release.json \
   NAME=hse HOST=hse.example.edu make vast-up
diff --git a/kernel/environments/base-gpu.txt b/kernel/environments/base-gpu.txt
index 0c6531c8..b584ef25 100644
--- a/kernel/environments/base-gpu.txt
+++ b/kernel/environments/base-gpu.txt
@@ -17,5 +17,11 @@
 # Обычный индекс pypi, а не .../whl/cpu, как в cv: именно там колёса с CUDA.
 # Драйвер приходит с хоста через nvidia-container-toolkit, ставить его в образ
 # не надо. Первая сборка идёт долго, дальше образ лежит в кеше docker.
-torch>=2.4
-torchvision>=0.19
+#
+# Pinned since 13 Oct 2026: an unpinned torch followed PyPI to CUDA 13.0 wheels
+# (torch 2.14.1+cu130), which need an NVIDIA driver of at least 580, while the
+# rental scripts still took hosts with CUDA 12 drivers, where torch imports and
+# never sees the card. A pin moves only on purpose, together with the driver
+# floor in scripts/vast-vm.sh and the kaggle-gpu pins checked on a card.
+torch==2.14.1
+torchvision==0.29.1
diff --git a/kernel/environments/kaggle-gpu.txt b/kernel/environments/kaggle-gpu.txt
new file mode 100644
index 00000000..3be5422b
--- /dev/null
+++ b/kernel/environments/kaggle-gpu.txt
@@ -0,0 +1,34 @@
+# Colloq: the kaggle-base stack on CUDA PyTorch, for neural networks in classes and competitions.
+# This is not the official Kaggle image.
+#
+# It is built on base-gpu (torch and torchvision as CUDA wheels) and inherits
+# its GPU flag. A room on it takes a card from KERNEL_GPUS; a competition on it
+# gets a share of a card only with its GPU switch on and cards given to
+# submissions in Resources. The card reaches the container through the NVIDIA
+# container runtime; where none is given (personal notebooks, a competition
+# with the switch off) the same image computes on the CPU.
+#
+# Above the kaggle-base pins: seaborn, which base-gpu lacks (it is a root, not
+# built on base), and OpenCV and timm for images. Built on the host like
+# base-gpu: publish.yml keeps GPU images out.
+# colloq: from base-gpu
+numpy==2.4.6
+pandas==3.0.6
+scipy==1.17.1
+scikit-learn==1.9.1
+catboost==1.2.10
+lightgbm==4.6.0
+xgboost==3.2.0
+optuna==4.9.0
+polars==1.44.2
+pyarrow==23.0.1
+statsmodels==0.15.0
+imbalanced-learn==0.14.2
+category-encoders==2.11.1
+shap==0.51.0
+h3==4.5.0
+nbclient==0.11.0
+nbformat==5.11.1
+seaborn==0.13.2
+opencv-python-headless==5.0.0.93
+timm==1.0.30
diff --git a/runtime/README.md b/runtime/README.md
index 21861cde..39610f88 100644
--- a/runtime/README.md
+++ b/runtime/README.md
@@ -49,6 +49,11 @@ Secret and catalog files are reread on requests, supporting atomic file updates.
 | `RUNTIME_GPU_RUNTIME_CLASS` | `nvidia` (empty: no RuntimeClass)                      |
 | `RUNTIME_GPU_NODE_SELECTOR` | `{}`                                                   |
 | `RUNTIME_GPU_TOLERATIONS`   | `[]`                                                   |
+| `RUNTIME_COMPETITION_GPU_RESOURCE` | unset (competition jobs get no GPU)             |
+| `RUNTIME_COMPETITION_GPU_COUNT` | `1`                                                |
+| `RUNTIME_COMPETITION_GPU_SLOTS` | unset (`null` in health: the app keeps one)        |
+| `RUNTIME_COMPETITION_GPU_SHM_MB` | `1024` (`0`: no `/dev/shm` volume, for MPS)       |
+| `RUNTIME_COMPETITION_GPU_PULL_SECONDS` | `900`                                       |
 | `RUNTIME_ROOM_NODE_SELECTOR`| `{}`                                                   |
 | `RUNTIME_ROOM_TOLERATIONS`  | `[]`                                                   |
 | `RUNTIME_PRIORITY_CLASS`    | unset                                                  |
@@ -64,6 +69,68 @@ a live resize refuses). The default must not exceed the ceiling. GPU environment
 memory is 64Mi; memory-backed volumes consume the Pod memory limit. The operator
 sets the node PID limit and namespace quota. PVC capacity is not a per-room quota.
 
+### Competition jobs on a GPU environment
+
+A GPU environment's image runs every competition job on the CPU: an inventory,
+a package preparation, the metric, and the notebook of a competition that asked
+for no GPU. Their `main` containers, and the egress preflight of every Pod of
+such an image, get `NVIDIA_VISIBLE_DEVICES=void`, so an image built from
+`nvidia/cuda` gets no card on a node whose default runtime is NVIDIA's. Only a
+notebook intent with `gpu: true` (the app sends it for a competition that asks
+for a GPU, and never for the metric, which mounts the hidden answers) gets one:
+
+- `RUNTIME_COMPETITION_GPU_COUNT` units (1 to 8) of
+  `RUNTIME_COMPETITION_GPU_RESOURCE` in the requests and limits of its `main`
+  container, never in the exporter or the preflight. The resource is an extended
+  resource name as the cluster's device plugin offers it: `nvidia.com/gpu` (a
+  whole card, or a time-sliced or MPS replica of one), `nvidia.com/gpu.shared`,
+  `nvidia.com/mig-1g.10gb`. What a unit is, the plugin's configuration decides;
+  the broker only asks for it.
+- The GPU rooms' placement: RuntimeClass `RUNTIME_GPU_RUNTIME_CLASS` (none when
+  empty), `RUNTIME_GPU_NODE_SELECTOR` over the rooms' selector, and
+  `RUNTIME_GPU_TOLERATIONS` after the rooms' tolerations.
+- A memory-backed `/dev/shm` of half the job's memory, at most
+  `RUNTIME_COMPETITION_GPU_SHM_MB` (1024), for PyTorch's DataLoader workers.
+  `0` gives the Pod no `/dev/shm` volume at all: with MPS sharing the device
+  plugin mounts its daemon's shared memory there, and the kubelet mounts a
+  Pod's own volumes after the plugin's, so ours would hide it and CUDA could
+  not reach the MPS server (NVIDIA/k8s-device-plugin#467). MPS setups set `0`.
+
+With `RUNTIME_COMPETITION_GPU_RESOURCE` unset such an intent is refused with 409
+before any Pod exists, and so is one for an environment whose catalog entry is
+not `gpu`. `/v1/health` reports `competition.gpu`: unavailable with code
+`gpu_unavailable` while the resource is unset, otherwise exactly what
+`competition.execution` says; and `competition.gpuSlots`:
+`RUNTIME_COMPETITION_GPU_SLOTS` (1 to 256, how many such notebooks of this
+instance the cluster runs at once, as the operator knows it), `null` when it
+is unset or the resource is. The broker only reports it: the app's queue keeps
+that many GPU notebooks at the broker, one for `null`, and lets CPU work go
+past the rest.
+
+A GPU notebook the scheduler cannot place (`Insufficient `), whose
+card the kubelet could not hand over (`UnexpectedAdmissionError`,
+`OutOf`), or whose Pod the API server refuses (HTTP 403, or a 400/422
+that says an admission check denied it: a ResourceQuota used up, a RuntimeClass
+the cluster lacks, an admission policy) fails as `unschedulable` with resource
+`gpu`, and the app returns it to its queue. For an API refusal `error` says
+which of the three, in the broker's words (a quota's resource names are the
+only thing taken from the API's message); the app shows it to the teacher.
+A refusal of the broker's own permissions, and any refusal of a job without a
+GPU, stays an error.
+
+### Image pull of a GPU environment
+
+Every Pod of a GPU environment's image (10 to 13 GB), whatever its job and
+whether or not it gets a GPU, has `RUNTIME_COMPETITION_GPU_PULL_SECONDS` (900; 0
+to 3600) to pull it onto a node that has never run it: the broker waits that
+much longer, after the 30 seconds of startup every job has, for `main` to
+start, its `activeDeadlineSeconds` grows by as much (annotation
+`colloq.dev/pull-seconds` keeps the Pod's own number), and so does the package
+proxy's life for a preparation. The job's limit still counts from `main`'s
+start, in the broker and in the notebook harness, which stops the notebook
+from its own start (`run_notebook.py` · `stop_at`); the app's wait for such a
+job counts from the moment it sees it running. A CPU image's Pods are as before.
+
 ### The customer's cluster
 
 The single-node k3s the installer builds needs none of the settings below; in a
@@ -81,9 +148,11 @@ variable's name in the message.
   an ensure keeps a live Pod with its current memory and CPU, and the room's next
   Pod gets the new numbers. `0`/`1` (`true`/`false` too).
 - `RUNTIME_ROOM_NODE_SELECTOR` (JSON object) and `RUNTIME_ROOM_TOLERATIONS` (JSON
-  array of Kubernetes tolerations) apply to room and personal-notebook Pods;
+  array of Kubernetes tolerations) apply to room and personal-notebook Pods, and
+  to competition job and resolver Pods, which run student code too;
   `RUNTIME_GPU_NODE_SELECTOR` / `RUNTIME_GPU_TOLERATIONS` are added for GPU rooms
-  only (a personal notebook never has a GPU). Competition Pods take neither.
+  and for a competition notebook given a GPU only (a personal notebook never has
+  a GPU). The package proxy takes neither.
 - `RUNTIME_PRIORITY_CLASS` is the priorityClassName of every broker Pod; a
   `system-*` class is refused.
 - `RUNTIME_POD_LABELS` / `RUNTIME_POD_ANNOTATIONS` (JSON objects of strings) are
@@ -93,19 +162,24 @@ variable's name in the message.
   refused. Use them for a policy engine's required labels, cost labels, or to opt
   the Pods out of sidecar injection (`sidecar.istio.io/inject: "false"`).
 
-On k3s, set `RUNTIME_KERNEL_MEMORY` and `RUNTIME_KERNEL_MEMORY_MAX` in the
-operator env file (`--env-file` of `cluster.sh install|prepare|update`, kept as
-`/var/lib/colloq/config.env`). `release.py render` copies only these two runtime
-keys into the broker Deployment, validates them with the same grammar and bounds
-as the broker (and an explicit ceiling against the default, `2Gi` when unset),
-and re-renders them on every update. An empty value means the broker default.
+On k3s, set `RUNTIME_KERNEL_MEMORY` and `RUNTIME_KERNEL_MEMORY_MAX`, and the
+`RUNTIME_COMPETITION_GPU_*` settings (`RESOURCE`, `COUNT`, `SLOTS`, `SHM_MB`,
+`PULL_SECONDS`), in the operator env file (`--env-file` of `cluster.sh
+install|prepare|update`, kept as `/var/lib/colloq/config.env`). `release.py
+render` copies only these seven runtime keys into the broker Deployment,
+validates them with the same grammar and bounds as the broker (and an explicit
+ceiling against the default, `2Gi` when unset), and re-renders them on every
+update. An empty value means the broker default.
 Other `RUNTIME_*` variables stay fixed by the installer. See
 [deploy/k3s/README.md](../deploy/k3s/README.md#room-memory).
 
 Kubernetes requests verify the configured CA and reread projected credentials
 on every request. Redirects are not followed. Requests have a 10-second timeout,
 256KiB request bound, and 2MiB response bound. Kubernetes error messages expose
-only status/reason, never arbitrary API response bodies.
+only status/reason, never arbitrary API response bodies; of a refusal's message
+the client keeps only the resource name a node lacked, and which kind of
+refusal it was (quota, with the quota's resource names; missing RuntimeClass;
+permission; admission).
 
 ## HTTP contract
 
@@ -116,7 +190,7 @@ private cluster network; use a secured transport boundary for remote access.
 
 | Method and path                    | Request                    | Response                                                         |
 | ---------------------------------- | -------------------------- | ---------------------------------------------------------------- |
-| GET `/v1/health`                   | —                          | `{ok, reason, defaultCpus?, defaultMemoryMb?, maxMemoryMb?, inPlaceResize?}`; 503 when unavailable |
+| GET `/v1/health`                   | —                          | `{ok, reason, defaultCpus?, defaultMemoryMb?, maxMemoryMb?, inPlaceResize?, competition: {execution, preparation, gpu, gpuSlots}}`; 503 when unavailable |
 | GET `/v1/catalog`                  | —                          | validated catalog                                                |
 | GET `/v1/rooms`                    | —                          | `{rooms: RuntimeRoom[]}`, personal-notebook Pods with `role: "own"` |
 | POST `/v1/rooms/:id`               | `{environment, revision?, cpus?, memoryMb?}` | `RuntimeEndpoint`                              |
diff --git a/runtime/src/competition-jobs.ts b/runtime/src/competition-jobs.ts
index e29434e5..d33038dd 100644
--- a/runtime/src/competition-jobs.ts
+++ b/runtime/src/competition-jobs.ts
@@ -1,8 +1,9 @@
 import { createHmac } from 'node:crypto'
 import { isIP } from 'node:net'
+import type { CompetitionCapabilities, RuntimeCapability } from '../../shared/capabilities.js'
 import { resolveRuntimeEnvironment, type RuntimeCatalog } from '../../shared/runtime.js'
 import {
-  COMPETITION_JOB_ID, parseCompetitionJobIntent,
+  COMPETITION_GPU_PULL_SECONDS_MAX, COMPETITION_JOB_ID, COMPETITION_RESOURCES_UNAVAILABLE, parseCompetitionJobIntent,
   type CompetitionJobCollection, type CompetitionJobIntent, type CompetitionJobKind,
   type CompetitionJobProgress, type CompetitionJobStatus,
 } from '../../shared/competition-runtime.js'
@@ -17,11 +18,22 @@ const jobName = (id: string) => `colloq-job-${id}`
 const proxyName = (id: string) => `colloq-proxy-${id}`
 const iso = (time: number) => new Date(time).toISOString()
 
-/** Where Pods that run student code go: the rooms' node selector and tolerations, when set. */
-function studentPlacement(config: CompetitionJobConfig): Record {
+/**
+ * Where Pods that run student code go: the rooms' node selector and
+ * tolerations, when set. A notebook given a GPU adds what a GPU room adds
+ * (controller.ts · placement): the GPU RuntimeClass, unless set empty, and the
+ * GPU nodes' selector over the students' and their tolerations after them.
+ * The cards a platform keeps for rooms are on those nodes; a submission's share
+ * of one is no different.
+ */
+function studentPlacement(config: CompetitionJobConfig, gpu = false): Record {
+  const runtimeClass = gpu ? config.gpuRuntimeClass ?? 'nvidia' : ''
+  const nodeSelector = { ...config.studentNodeSelector, ...(gpu ? config.gpuNodeSelector : {}) }
+  const tolerations = [...(config.studentTolerations ?? []), ...(gpu ? config.gpuTolerations ?? [] : [])]
   return {
-    ...(config.studentNodeSelector && Object.keys(config.studentNodeSelector).length ? { nodeSelector: config.studentNodeSelector } : {}),
-    ...(config.studentTolerations?.length ? { tolerations: config.studentTolerations } : {}),
+    ...(runtimeClass ? { runtimeClassName: runtimeClass } : {}),
+    ...(Object.keys(nodeSelector).length ? { nodeSelector } : {}),
+    ...(tolerations.length ? { tolerations } : {}),
   }
 }
 
@@ -54,6 +66,38 @@ export interface CompetitionJobConfig extends PodPolicy {
   /** RUNTIME_ROOM_NODE_SELECTOR / RUNTIME_ROOM_TOLERATIONS, for the Pods that run student code. */
   studentNodeSelector?: Record
   studentTolerations?: Toleration[]
+  /**
+   * RUNTIME_COMPETITION_GPU_RESOURCE / _COUNT: what the main container of a
+   * notebook job whose intent asks for a GPU requests, in its requests and its
+   * limits alike (an extended resource is never overcommitted). Absent: no
+   * competition job gets a GPU, and one that asks is refused.
+   *
+   * Every other job runs a GPU environment's image on the CPU: an inventory, a
+   * package preparation, the metric (it mounts the hidden answers and must
+   * never share a device with a participant's code), and a notebook whose
+   * competition asked for no card.
+   *
+   * `slots` (RUNTIME_COMPETITION_GPU_SLOTS): how many such notebooks the
+   * cluster runs at once for this instance, as the operator knows it; only
+   * reported (health), for the app's queue. `shmMb`
+   * (RUNTIME_COMPETITION_GPU_SHM_MB): the ceiling of the notebook's own
+   * /dev/shm, 1024 when absent, 0 for none (shmMb below).
+   */
+  gpu?: { resource: string; count: number; slots?: number | null; shmMb?: number }
+  /**
+   * RUNTIME_COMPETITION_GPU_PULL_SECONDS: the pull allowance of every Pod of a
+   * GPU environment's image, whether or not it gets a card; 900 when absent
+   * (pullSeconds below).
+   */
+  gpuPullSeconds?: number
+  /**
+   * RUNTIME_GPU_RUNTIME_CLASS / _NODE_SELECTOR / _TOLERATIONS: the GPU rooms'
+   * placement, which a notebook given a GPU shares. As for rooms, an absent
+   * RuntimeClass is `nvidia` and an empty one is none.
+   */
+  gpuRuntimeClass?: string
+  gpuNodeSelector?: Record
+  gpuTolerations?: Toleration[]
 }
 interface Options {
   kube: KubernetesClient
@@ -113,6 +157,43 @@ while True:
 function packageRoom(intent: CompetitionJobIntent): number | null {
   return intent.kind === 'notebook' && intent.bundleId && intent.limits.packagesMb ? intent.limits.packagesMb : null
 }
+/**
+ * The annotation of a job Pod that holds its image pull allowance, seconds
+ * (CompetitionJobs · pullSeconds): the Pod's own record, so that its clock
+ * stays the one it was created with across a broker restart with other
+ * settings.
+ */
+const PULL_SECONDS = 'colloq.dev/pull-seconds'
+/** The pull allowance a job Pod was created with; 0 for a CPU image's Pod and for one from a broker before it. */
+function pullAllowance(pod: KubeObject): number {
+  const seconds = Number(pod.metadata.annotations?.[PULL_SECONDS])
+  return Number.isSafeInteger(seconds) && seconds > 0 ? Math.min(seconds, COMPETITION_GPU_PULL_SECONDS_MAX) : 0
+}
+/**
+ * For every container of a GPU environment's image that requests no GPU: the
+ * metric, an inventory, a preparation, a notebook without a card, the egress
+ * preflight. Images built from nvidia/cuda carry NVIDIA_VISIBLE_DEVICES=all,
+ * and on a node whose default container runtime is NVIDIA's (the GPU
+ * operator's default on many clusters) that alone mounts every card of the
+ * node into the container, whatever the scheduler counted. `void` is the
+ * runtime's own word for "no GPU and no driver at all"; any other runtime
+ * ignores the variable. Never on the main container of a notebook given a
+ * GPU: there the device plugin sets it to the units it allocated, and ours
+ * would override it.
+ */
+const noGpu = () => env('NVIDIA_VISIBLE_DEVICES', 'void')
+/**
+ * The resource names a scheduler refusal says it lacked: "0/3 nodes are
+ * available: 1 Insufficient nvidia.com/gpu, 2 Insufficient memory." gives
+ * `nvidia.com/gpu` and `memory`. Only the names, as for rooms
+ * (controller.ts · unschedulable): the rest is about other nodes and other
+ * Pods. A name ends with a letter or a digit, so a dot after it is the end of
+ * the scheduler's sentence.
+ */
+function shortages(message: string): Set {
+  return new Set([...message.slice(0, 4096).matchAll(/\binsufficient\s+([a-z0-9](?:[a-z0-9._/-]*[a-z0-9])?)/gi)]
+    .map((match) => match[1].toLowerCase()))
+}
 function exportBudget(kind: CompetitionJobKind, target: number): number {
   if (kind === 'notebook') return Math.min(2 * target, 256 * MiB)
   if (kind === 'metric') return Math.min(2 * target, 16 * MiB)
@@ -137,27 +218,52 @@ export class CompetitionJobs {
     if (!IMAGE.test(exporterImage) || !/^[a-z0-9-]{1,40}$/.test(instanceId) || !/^[a-z0-9-]{1,63}$/.test(dataClaim))
       throw new RuntimeError('Competition broker is not configured', 503)
   }
-  async health(): Promise {
+  async health(): Promise {
     const unavailable = (code: string, reason: string) => ({ available: false, code, reason })
     const available = { available: true, code: 'ready', reason: null }
+    /*
+     * `gpu`: whether a notebook that asks for a card can run. Only where the
+     * operator named the cluster's GPU resource, and then exactly when
+     * execution can: a GPU job is a job first, with the same claim, isolation
+     * and images. The app refuses a GPU competition's uploads and baseline
+     * check at the door on this answer, instead of queueing work this broker
+     * would refuse.
+     *
+     * `gpuSlots`: how many such notebooks the operator says the cluster runs
+     * at once (RUNTIME_COMPETITION_GPU_SLOTS), null when unset or off. The
+     * app's queue keeps that many at the broker, one for null, and lets CPU
+     * work go past the others: a GPU notebook Kubernetes cannot place holds
+     * a general slot for the 30 seconds this broker waits on the scheduler
+     * (phase · unschedulable), and a queue that sent every waiting one would
+     * keep every slot busy with them.
+     */
+    const off = unavailable('gpu_unavailable', 'Competition jobs get no GPU on this cluster: RUNTIME_COMPETITION_GPU_RESOURCE (Helm: gpu.competitions.resource) is not set')
+    const answer = (execution: RuntimeCapability, preparation: RuntimeCapability): CompetitionCapabilities =>
+      ({ execution, preparation, gpu: this.options.config.gpu ? execution : off, gpuSlots: this.options.config.gpu?.slots ?? null })
     try { this.configured() }
-    catch { const state = unavailable('broker_unavailable', 'Competition broker configuration is incomplete'); return { execution: state, preparation: state } }
+    catch { const state = unavailable('broker_unavailable', 'Competition broker configuration is incomplete'); return answer(state, state) }
+    /*
+     * Any pinned image will do. This used to ask for an image without a GPU,
+     * because a GPU environment's image was refused for every job; now it runs
+     * every kind of job on the CPU, and only a notebook that asked for a card
+     * gets one, so a catalog of GPU environments alone is a working catalog.
+     */
     let hasImage = false
-    try { hasImage = this.options.catalog().environments.some(row => IMAGE.test(row.image) && !row.gpu) } catch { /* reported below */ }
+    try { hasImage = this.options.catalog().environments.some(row => IMAGE.test(row.image)) } catch { /* reported below */ }
     if (!hasImage) {
       const state = unavailable('image_unavailable', 'No pinned competition registry image is available')
-      return { execution: state, preparation: state }
+      return answer(state, state)
     }
     try { await this.options.kube.request('GET', `${this.root}/persistentvolumeclaims/${this.options.config.dataClaim}`) }
-    catch { const state = unavailable('storage_unavailable', 'Competition data claim is unavailable'); return { execution: state, preparation: state } }
+    catch { const state = unavailable('storage_unavailable', 'Competition data claim is unavailable'); return answer(state, state) }
     const policy = async (name: string) => this.options.kube.request('GET', `/apis/networking.k8s.io/v1/namespaces/${this.options.config.namespace}/networkpolicies/${name}`)
     try { await policy('competition-job-isolation') }
-    catch { const state = unavailable('isolation_unavailable', 'Competition job network isolation is unavailable'); return { execution: state, preparation: state } }
-    let preparation: import('../../shared/capabilities.js').RuntimeCapability = available
+    catch { const state = unavailable('isolation_unavailable', 'Competition job network isolation is unavailable'); return answer(state, state) }
+    let preparation: RuntimeCapability = available
     try { await policy('competition-resolver-isolation'); await policy('competition-proxy-isolation');
       await this.options.kube.request('GET', `${this.root}/services?limit=1`) }
     catch { preparation = unavailable('preparation_unavailable', 'Dependency resolver proxy isolation is unavailable') }
-    return { execution: available, preparation }
+    return answer(available, preparation)
   }
   private labels(jobId: string, role: string) {
     return { 'app.kubernetes.io/managed-by': MANAGER, 'colloq.dev/instance': this.options.config.instanceId,
@@ -173,7 +279,60 @@ export class CompetitionJobs {
     const attempt = `competitions/${intent.competitionId}/s/${intent.submissionId}/attempts/${intent.attemptId}`
     return `${attempt}/${intent.kind === 'metric' ? 'score-out' : 'result'}`
   }
-  private main(intent: CompetitionJobIntent, image: string) {
+  /**
+   * The GPU this job's Pod requests, or null: a notebook intent with
+   * `gpu: true` on a broker that names the cluster's GPU resource. Anything
+   * else that asks for one never reaches a Pod (startPrepared).
+   */
+  private gpu(intent: CompetitionJobIntent): NonNullable | null {
+    return intent.kind === 'notebook' && intent.gpu === true && this.options.config.gpu ? this.options.config.gpu : null
+  }
+  /**
+   * /dev/shm of a notebook given a GPU, MiB; 0 — none of ours. PyTorch's
+   * DataLoader workers hand their batches over through it, and the 64 MiB a
+   * container has by default ends such a run with "bus error". Memory-backed,
+   * and so paid for from the job's memory like every such volume: half of it
+   * at most, and never more than the operator's ceiling
+   * (RUNTIME_COMPETITION_GPU_SHM_MB), by default the 1 GiB a GPU room gets
+   * (controller.ts).
+   *
+   * A ceiling of 0 is for MPS sharing: there the device plugin mounts its MPS
+   * daemon's shared memory at /dev/shm for the CUDA client to reach the
+   * server, and the kubelet mounts the Pod's own volumes after the plugin's,
+   * so a volume of ours at the same path hides it: CUDA fails to start, and
+   * the notebook trains on the CPU until its limit or dies in its first
+   * `.cuda()` (NVIDIA/k8s-device-plugin#467, #805). The /dev/shm is then the
+   * daemon's, shared by the node's MPS clients and sized by it.
+   */
+  private shmMb(intent: CompetitionJobIntent): number {
+    const gpu = this.gpu(intent)
+    return gpu ? Math.min(gpu.shmMb ?? 1024, Math.floor(intent.limits.memoryMb / 2)) : 0
+  }
+  /**
+   * The pull allowance of a Pod of this image, seconds: 0 for a CPU image, and
+   * for a GPU environment's image RUNTIME_COMPETITION_GPU_PULL_SECONDS (900
+   * when unset).
+   *
+   * Such an image is 10 to 13 GB, and every job of a GPU competition runs it,
+   * the metric and a package preparation included, on whatever node the
+   * scheduler picks: a metric on a CPU node that never ran it, a notebook on a
+   * GPU node that has not yet. Its pull happens after the Pod is created and
+   * before its first container starts, and without the allowance it came out
+   * of the job's own time: 30 seconds of startup and then the wall limit of
+   * the job (a metric's is two minutes), so the first metric on every cold
+   * node failed as "Wall time limit exceeded", and a notebook lost to the pull
+   * what it had to train. With it, the broker waits that much longer for
+   * `main` to start (phase), the Pod's activeDeadlineSeconds grows by as
+   * much, and so does the package proxy's life for a preparation. A started
+   * job still gets exactly its limit from its own start: the broker counts
+   * the wall from `main`'s start, and the notebook's harness stops it from
+   * its own (run_notebook.py · stop_at), so the allowance only ever covers
+   * the wait before.
+   */
+  private pullSeconds(gpuImage: boolean): number {
+    return gpuImage ? Math.min(this.options.config.gpuPullSeconds ?? 900, COMPETITION_GPU_PULL_SECONDS_MAX) : 0
+  }
+  private main(intent: CompetitionJobIntent, image: string, gpuImage: boolean) {
     const base = `competitions/${intent.competitionId}/s/${intent.submissionId}`
     const attempt = `${base}/attempts/${intent.attemptId}`
     // With the hidden test the app has composed this attempt's own data folder
@@ -195,10 +354,14 @@ export class CompetitionJobs {
           : `/harness/broker_prepare.py ${intent.kind}`
     // Entrant stdout can be unbounded. Reports live in the bounded memory volume.
     const command = ['sh', '-c', `exec python -I -u ${script} >/dev/null 2>&1`]
+    // The units of the cluster's GPU resource, in requests and limits alike:
+    // Kubernetes allows an extended resource no other way.
+    const gpu = this.gpu(intent)
+    const card = gpu ? { [gpu.resource]: gpu.count } : {}
     return { name: 'main', image, imagePullPolicy: 'IfNotPresent', command,
       workingDir: '/work', securityContext: security,
-      resources: { requests: { cpu: `${intent.limits.cpus}`, memory: `${intent.limits.memoryMb}Mi`, 'ephemeral-storage': '64Mi' },
-        limits: { cpu: `${intent.limits.cpus}`, memory: `${intent.limits.memoryMb}Mi`, 'ephemeral-storage': '256Mi' } },
+      resources: { requests: { cpu: `${intent.limits.cpus}`, memory: `${intent.limits.memoryMb}Mi`, 'ephemeral-storage': '64Mi', ...card },
+        limits: { cpu: `${intent.limits.cpus}`, memory: `${intent.limits.memoryMb}Mi`, 'ephemeral-storage': '256Mi', ...card } },
       env: [env('HOME', '/tmp/home'), env('PYTHONDONTWRITEBYTECODE', '1'), env('PYTHONNOUSERSITE', '1'),
         env('PYTHONUNBUFFERED', '1'), env('PIP_CONFIG_FILE', '/dev/null'),
         // Every thread pool a notebook is likely to meet, capped at the Pod's
@@ -217,25 +380,34 @@ export class CompetitionJobs {
         ...(packageRoom(intent) ? [env('COMP_PACKAGES', '/packages')] : []),
         ...(intent.kind === 'resolve' ? [env('HTTPS_PROXY', `http://${proxyName(intent.jobId)}:3128`), env('HTTP_PROXY', `http://${proxyName(intent.jobId)}:3128`),
           env('https_proxy', `http://${proxyName(intent.jobId)}:3128`), env('http_proxy', `http://${proxyName(intent.jobId)}:3128`),
-          env('NO_PROXY', ''), env('no_proxy', ''), env('ALL_PROXY', ''), env('all_proxy', '')] : [])],
+          env('NO_PROXY', ''), env('no_proxy', ''), env('ALL_PROXY', ''), env('all_proxy', '')] : []),
+        ...(gpuImage && !gpu ? [noGpu()] : [])],
       volumeMounts: [mount('out', '/out'), mount('work', '/work'), mount('tmp', '/tmp'),
         ...(packageRoom(intent) ? [mount('packages', '/packages')] : []),
+        ...(this.shmMb(intent) ? [mount('shm', '/dev/shm')] : []),
         mount('data', '/harness', intent.kind === 'notebook' || intent.kind === 'metric' ? 'competitions/harness' : 'dependencies/harness', true), ...input],
     }
   }
-  private pod(intent: CompetitionJobIntent, image: string): KubeObject {
+  private pod(intent: CompetitionJobIntent, image: string, gpuImage: boolean): KubeObject {
     const role = intent.kind === 'resolve' ? 'competition-resolver' : 'competition-job'
+    const gpu = this.gpu(intent)
+    const pull = this.pullSeconds(gpuImage)
+    const shm = this.shmMb(intent)
     return { apiVersion: 'v1', kind: 'Pod', metadata: { name: jobName(intent.jobId),
-        ...policyMetadata(this.options.config, this.labels(intent.jobId, role), this.annotations(intent)) },
+        ...policyMetadata(this.options.config, this.labels(intent.jobId, role),
+          { ...this.annotations(intent), ...(pull ? { [PULL_SECONDS]: String(pull) } : {}) }) },
       spec: { restartPolicy: 'Never', automountServiceAccountToken: false, enableServiceLinks: false,
-        terminationGracePeriodSeconds: 5, activeDeadlineSeconds: intent.limits.wallSeconds + 90,
+        // Counted by the kubelet from the Pod's start, the image pull included.
+        terminationGracePeriodSeconds: 5, activeDeadlineSeconds: intent.limits.wallSeconds + 90 + pull,
         securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 },
         ...(this.options.config.imagePullSecret ? { imagePullSecrets: [{ name: this.options.config.imagePullSecret }] } : {}),
         // Both of this Pod's containers mount the data claim.
         ...policyPlacement(this.options.config, true),
-        ...studentPlacement(this.options.config),
+        ...studentPlacement(this.options.config, gpu !== null),
         ...(intent.kind === 'resolve' ? {} : { initContainers: [{ name: 'egress-probe', image,
           imagePullPolicy: 'IfNotPresent', command: ['python', '-I', '-c', OFFLINE_PROBE],
+          // The preflight runs the job's image, but never needs its card.
+          ...(gpuImage ? { env: [noGpu()] } : {}),
           securityContext: security,
           resources: { requests: { cpu: '50m', memory: '64Mi' }, limits: { cpu: '250m', memory: '128Mi', 'ephemeral-storage': '16Mi' } },
         }] }),
@@ -248,8 +420,9 @@ export class CompetitionJobs {
           // Never more than 256 MiB, whatever the job: a package preparation works
           // in /work (COMP_OUT), which the app sizes for the phase through tmpfsMb.
           { name: 'tmp', emptyDir: memory(Math.min(intent.limits.tmpfsMb, 256)) },
-          ...(packageRoom(intent) ? [{ name: 'packages', emptyDir: memory(packageRoom(intent)!) }] : [])],
-        containers: [this.main(intent, image), { name: 'exporter', image: this.options.config.exporterImage,
+          ...(packageRoom(intent) ? [{ name: 'packages', emptyDir: memory(packageRoom(intent)!) }] : []),
+          ...(shm ? [{ name: 'shm', emptyDir: memory(shm) }] : [])],
+        containers: [this.main(intent, image, gpuImage), { name: 'exporter', image: this.options.config.exporterImage,
           imagePullPolicy: 'IfNotPresent', command: ['node', '/app/export-sidecar.js'], securityContext: security,
           resources: { requests: { cpu: '100m', memory: '64Mi' }, limits: { cpu: '500m', memory: '256Mi', 'ephemeral-storage': '64Mi' } },
           env: [env('COMP_EXPORT_TOKEN', this.token(intent.jobId)), env('COMP_EXPORT_KIND', intent.kind),
@@ -261,11 +434,17 @@ export class CompetitionJobs {
         }],
       } }
   }
-  private proxyPod(intent: CompetitionJobIntent): KubeObject {
+  /**
+   * `pull`: the resolver's image pull allowance (pullSeconds). The proxy is
+   * up before the resolver's Pod is created, and its own clock
+   * (COMP_PROXY_WALL_SECONDS) runs from its start: a resolver that spends its
+   * allowance pulling still has the proxy for its whole limit.
+   */
+  private proxyPod(intent: CompetitionJobIntent, pull = 0): KubeObject {
     return { apiVersion: 'v1', kind: 'Pod', metadata: { name: proxyName(intent.jobId),
         ...policyMetadata(this.options.config, this.labels(intent.jobId, 'competition-proxy'), this.annotations(intent)) },
       spec: { restartPolicy: 'Never', automountServiceAccountToken: false, enableServiceLinks: false,
-        activeDeadlineSeconds: intent.limits.wallSeconds + 90, securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000 },
+        activeDeadlineSeconds: intent.limits.wallSeconds + 90 + pull, securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000 },
         // No claim mounted: the priority, but no colocation.
         ...policyPlacement(this.options.config, false),
         containers: [{ name: 'proxy', image: this.options.config.exporterImage, command: ['node', '/app/competition-proxy.js'],
@@ -274,7 +453,7 @@ export class CompetitionJobs {
           livenessProbe: SERVER_LIVENESS(3128),
           resources: { requests: { cpu: '100m', memory: '64Mi' }, limits: { cpu: '500m', memory: '128Mi', 'ephemeral-storage': '32Mi' } },
           env: [env('COMP_PROXY_MAX_BYTES', String(Math.min(intent.limits.targetBytes * 3 + 32 * 1024 * 1024, 1536 * 1024 * 1024))),
-            env('COMP_PROXY_WALL_SECONDS', String(intent.limits.wallSeconds)),
+            env('COMP_PROXY_WALL_SECONDS', String(intent.limits.wallSeconds + pull)),
             ...(this.options.config.dependencyIndexUrl ? [env('DEPENDENCY_INDEX_URL', this.options.config.dependencyIndexUrl)] : []),
             ...(this.options.config.dependencyFilesHosts ? [env('DEPENDENCY_FILES_HOSTS', this.options.config.dependencyFilesHosts)] : [])],
           volumeMounts: [{ name: 'tmp', mountPath: '/tmp' }],
@@ -328,15 +507,28 @@ export class CompetitionJobs {
   private async startPrepared(intent: CompetitionJobIntent): Promise {
     const jobId = intent.jobId
     let image: string
-    let gpu = false
+    let gpuImage = false
     try {
       const environment = resolveRuntimeEnvironment(this.options.catalog(), intent.environment, intent.revision)
       image = environment.image
-      gpu = environment.gpu
+      gpuImage = environment.gpu
     }
     catch { throw new RuntimeError('Competition image revision is unavailable in catalog', 409) }
     if (!IMAGE.test(image)) throw new RuntimeError('Competition image is not a pinned registry digest', 409)
-    if (gpu) throw new RuntimeError('GPU competition images are unsupported by this executor', 409)
+    /*
+     * A GPU environment's image runs any job on the CPU, as a personal
+     * notebook's Pod runs it (controller.ts · pod). A card goes only to a
+     * notebook whose intent asks for one, and only where the operator named
+     * the cluster's GPU resource. Asked for anywhere else, it is refused here,
+     * before a Pod exists: training on the CPU without a word to anyone is
+     * what GPU competitions did before they could get a card, and it is not
+     * an answer.
+     */
+    if (intent.gpu === true) {
+      if (!this.options.config.gpu)
+        throw new RuntimeError('Competition GPU jobs are off on this broker: RUNTIME_COMPETITION_GPU_RESOURCE is not set', 409)
+      if (!gpuImage) throw new RuntimeError('A competition GPU job needs a GPU environment from the catalog', 409)
+    }
     const readiness = await this.health()
     const capability = intent.kind === 'resolve' || intent.kind === 'verify' ? readiness.preparation : readiness.execution
     if (!capability.available) throw new RuntimeError(capability.reason ?? 'Competition executor is unavailable', 503)
@@ -349,7 +541,7 @@ export class CompetitionJobs {
     }
     if (intent.kind === 'resolve') {
       await this.cleanupProxy(jobId)
-      const proxy = this.proxyPod(intent)
+      const proxy = this.proxyPod(intent, this.pullSeconds(gpuImage))
       const service = this.proxyService(intent)
       await this.options.kube.request('POST', `${this.root}/pods`, proxy)
       try { await this.options.kube.request('POST', `${this.root}/services`, service); await this.awaitProxy(jobId) }
@@ -357,11 +549,50 @@ export class CompetitionJobs {
     }
     if (this.cancelled.has(jobId)) { if (intent.kind === 'resolve') await this.cleanupProxy(jobId); throw new RuntimeError('Competition job was cancelled', 409) }
     let created: KubeObject
-    try { created = await this.options.kube.request('POST', `${this.root}/pods`, this.pod(intent, image)) }
-    catch (err) { if (intent.kind === 'resolve') await this.cleanupProxy(jobId).catch(() => {}); throw err }
+    try { created = await this.options.kube.request('POST', `${this.root}/pods`, this.pod(intent, image, gpuImage)) }
+    catch (err) {
+      if (intent.kind === 'resolve') await this.cleanupProxy(jobId).catch(() => {})
+      const refused = this.refusedAtAdmission(intent, err)
+      if (refused) return refused
+      throw err
+    }
     if (this.cancelled.has(jobId)) throw new RuntimeError('Competition job was cancelled', 409)
     return this.phase(created, intent)
   }
+  /**
+   * The API server's "no" to the Pod of a notebook given a GPU, as the status
+   * of a job that found no room: `unschedulable`, resource `gpu`, and the
+   * cause in words of our own for the teacher, while the app puts the job
+   * back into its queue and tries again a few seconds to a minute later.
+   *
+   * The Pod is refused before it exists when the namespace's ResourceQuota
+   * has no unit of the GPU resource left (GPU rooms and the submissions
+   * already running hold them: it frees itself), when the cluster has no
+   * RuntimeClass of the name RUNTIME_GPU_RUNTIME_CLASS gives (the chart's
+   * `nvidia`, absent on a cluster without the GPU operator), or when an
+   * admission policy turns down what only a GPU notebook's Pod carries: the
+   * units, the RuntimeClass, the GPU nodes' tolerations. It used to come back
+   * as an error, which the app records as a failed metric: every GPU
+   * submission of the class "failed" within seconds, nothing retried them,
+   * and the teacher read only "Forbidden". As with the scheduler and the
+   * kubelet (phase · cardRefused), it is "no card for this job now".
+   *
+   * Only for a GPU notebook: a CPU job refused at admission is not waiting
+   * for a card and stays the error it was. Nor a refusal of the broker's own
+   * permissions (RBAC), which every job would meet.
+   */
+  private refusedAtAdmission(intent: CompetitionJobIntent, err: unknown): CompetitionJobStatus | null {
+    const refusal = this.gpu(intent) && err instanceof KubernetesError ? err.refusal : undefined
+    if (!refusal || refusal.kind === 'permission') return null
+    const error = refusal.kind === 'quota'
+      ? `Kubernetes refused the GPU notebook's Pod: the namespace's resource quota is used up${refusal.resources.length ? ` (${refusal.resources.join(', ')})` : ''}`
+      : refusal.kind === 'runtime-class'
+        ? `Kubernetes refused the GPU notebook's Pod: the cluster has no RuntimeClass "${this.options.config.gpuRuntimeClass ?? 'nvidia'}" (RUNTIME_GPU_RUNTIME_CLASS, Helm: gpu.runtimeClassName)`
+        : "Kubernetes refused the GPU notebook's Pod at admission: a policy of the cluster denied it"
+    const now = this.now()
+    return { jobId: intent.jobId, kind: intent.kind, phase: 'failed', startedAt: now, finishedAt: now, exitCode: null, oomKilled: false,
+      progress: null, error: error.slice(0, 300), failure: { code: 'unschedulable', resource: 'gpu' } }
+  }
   private parseIntent(pod: KubeObject): CompetitionJobIntent {
     try { return parseCompetitionJobIntent(JSON.parse(pod.metadata.annotations?.['colloq.dev/intent'] ?? '')) }
     catch { throw new RuntimeError('Competition Pod has invalid intent metadata', 409) }
@@ -372,8 +603,11 @@ export class CompetitionJobs {
     const terminated = main?.state?.terminated
     const mainStartedAt = Date.parse(main?.state?.running?.startedAt ?? terminated?.startedAt ?? '')
     // Image startup and policy propagation can consume 30 seconds before main
-    // starts. Kubernetes reports the main start time once it has actually run.
-    const wallStartedAt = Number.isFinite(mainStartedAt) ? mainStartedAt : startedAt + (terminated ? 0 : 30_000)
+    // starts, and a GPU environment's image its pull allowance on top
+    // (pullSeconds). Kubernetes reports the main start time once it has
+    // actually run, and the limit counts from there.
+    const wallStartedAt = Number.isFinite(mainStartedAt) ? mainStartedAt
+      : startedAt + (terminated ? 0 : 30_000 + pullAllowance(pod) * 1000)
     const wallDeadline = wallStartedAt + intent.limits.wallSeconds * 1000
     const timedOut = this.now() > wallDeadline
     const scheduling = Array.isArray(pod.status?.conditions) ? pod.status.conditions.find((condition: any) =>
@@ -395,18 +629,33 @@ export class CompetitionJobs {
       imageUnavailable && !collected || exporterFailed) ? 'failed' :
       terminated ? collected ? exitCode === 0 ? 'complete' : 'failed' : 'exporting' : main?.state?.running ? 'running' : 'queued'
     const schedulerMessage = typeof scheduling?.message === 'string' ? scheduling.message : ''
-    const resource = /insufficient\s+memory/i.test(schedulerMessage) ? 'memory' :
+    // The card first, as for rooms (controller.ts · SHORTAGES): advice to ask
+    // for less memory does nothing for a job that did not get a card.
+    const gpu = this.gpu(intent)
+    const resource = gpu && shortages(schedulerMessage).has(gpu.resource.toLowerCase()) ? 'gpu' :
+      /insufficient\s+memory/i.test(schedulerMessage) ? 'memory' :
       /insufficient\s+cpu/i.test(schedulerMessage) ? 'cpu' :
         /insufficient\s+ephemeral-storage/i.test(schedulerMessage) ? 'storage' : 'other'
+    /*
+     * The kubelet can still refuse a GPU Pod the scheduler has placed: the
+     * device plugin failed to hand the units over (UnexpectedAdmissionError;
+     * a card whose previous job is still being torn down is a typical cause),
+     * or the node has none left after all (OutOf). For the job that
+     * is the same "no card free right now" as a scheduler refusal, and the
+     * same answer: back to the queue, not a failed metric.
+     */
+    const cardRefused = !!gpu && failedPod && !terminated &&
+      (pod.status?.reason === 'UnexpectedAdmissionError' || pod.status?.reason === `OutOf${gpu.resource}`)
     const failure: CompetitionJobStatus['failure'] | undefined = deadlineExceeded ? undefined :
       unschedulable ? { code: 'unschedulable', resource } :
+      cardRefused ? { code: 'unschedulable', resource: 'gpu' } :
       imageUnavailable ? { code: 'image_unavailable' } :
         exporterFailed ? { code: 'export_failed' } :
           (failedPod && pod.status?.reason !== 'DeadlineExceeded' || preflightFailed) && !terminated ? { code: 'runtime_unavailable' } : undefined
     return { jobId: intent.jobId, kind: intent.kind, phase, startedAt,
       finishedAt: deadlineExceeded ? this.now() : terminated ? mainFinishedAt || this.now() : phase === 'failed' ? this.now() : null,
       exitCode, oomKilled: !deadlineExceeded && terminated?.reason === 'OOMKilled', progress: null,
-      error: deadlineExceeded ? 'Wall time limit exceeded' : preflightFailed ? 'Network isolation preflight failed' : unschedulable ? 'Competition resources are unavailable' :
+      error: deadlineExceeded ? 'Wall time limit exceeded' : preflightFailed ? 'Network isolation preflight failed' : unschedulable || cardRefused ? COMPETITION_RESOURCES_UNAVAILABLE :
         imageUnavailable ? 'Competition image is unavailable' : exporterFailed ? 'Competition exporter failed' :
         failedPod && !terminated ? 'Competition Pod failed before execution' :
             terminated?.reason === 'OOMKilled' ? 'Memory limit exceeded' : null,
@@ -545,7 +794,8 @@ export class CompetitionJobs {
         jobName(id))) continue
       const age = this.now() - (Number(pod.metadata.annotations?.['colloq.dev/started-at']) || this.now())
       let ttl: number
-      try { ttl = Math.max(2 * 3600_000, this.parseIntent(pod).limits.wallSeconds * 1000 + 600_000) }
+      // A GPU image's pull allowance is the job's time as much as its limit is.
+      try { ttl = Math.max(2 * 3600_000, (this.parseIntent(pod).limits.wallSeconds + pullAllowance(pod)) * 1000 + 600_000) }
       catch { continue }
       if (age >= ttl) await this.cancel(id)
     }
diff --git a/runtime/src/config.ts b/runtime/src/config.ts
index 4869ab53..5c9c1876 100644
--- a/runtime/src/config.ts
+++ b/runtime/src/config.ts
@@ -1,7 +1,9 @@
 import { readFileSync, statSync } from 'node:fs'
 import { totalmem } from 'node:os'
 import { parseRuntimeCatalog, type RuntimeCatalog } from '../../shared/runtime.js'
+import { COMPETITION_GPU_PULL_SECONDS_MAX, COMPETITION_GPU_SLOTS_MAX } from '../../shared/competition-runtime.js'
 import {
+  parseExtendedResource,
   parsePriorityClass,
   parseRuntimeClass,
   parseStringMap,
@@ -65,6 +67,38 @@ export interface RuntimeConfig extends WorkloadConfig {
   competitionExporterImage: string
   competitionInstanceId: string
   competitionDataClaim: string
+  /**
+   * RUNTIME_COMPETITION_GPU_RESOURCE: what the notebook Pod of a competition
+   * that asks for a GPU requests, as the cluster's device plugin names it
+   * (pod-policy.ts · parseExtendedResource); empty is off, and such a
+   * notebook is refused. What one unit is (a whole card, a time-sliced or MPS
+   * replica, a MIG slice) is the device plugin's configuration, not ours.
+   */
+  competitionGpuResource: string
+  /** RUNTIME_COMPETITION_GPU_COUNT: how many units of it, 1 to 8; 1 when unset. */
+  competitionGpuCount: number
+  /**
+   * RUNTIME_COMPETITION_GPU_SLOTS: how many such notebooks of this instance
+   * the cluster runs at once, 1 to 256, as the operator knows it (the units
+   * set aside for submissions over the count); null when unset. The broker
+   * only reports it (/v1/health · competition.gpuSlots): the app's queue
+   * keeps that many GPU notebooks at the broker, one when it is null, and
+   * lets CPU work go past the rest.
+   */
+  competitionGpuSlots: number | null
+  /**
+   * RUNTIME_COMPETITION_GPU_SHM_MB: the ceiling of such a notebook's own
+   * memory-backed /dev/shm, MiB, 0 to 65536; 1024 when unset. 0 gives it
+   * none, for MPS sharing, whose device plugin mounts its daemon's shared
+   * memory there (competition-jobs.ts · shmMb).
+   */
+  competitionGpuShmMb: number
+  /**
+   * RUNTIME_COMPETITION_GPU_PULL_SECONDS: how long a Pod of a GPU
+   * environment's image may take to pull it before the job's own time starts,
+   * 0 to 3600; 900 when unset (competition-jobs.ts · pullSeconds).
+   */
+  competitionGpuPullSeconds: number
   dependencyIndexUrl: string
   dependencyFilesHosts: string
 }
@@ -182,6 +216,27 @@ export function loadRuntimeConfig(
   const roomTolerations = parseTolerations('RUNTIME_ROOM_TOLERATIONS', env.RUNTIME_ROOM_TOLERATIONS)
   const gpuNodeSelector = parseStringMap('RUNTIME_GPU_NODE_SELECTOR', env.RUNTIME_GPU_NODE_SELECTOR, 'nodeSelector')
   const gpuTolerations = parseTolerations('RUNTIME_GPU_TOLERATIONS', env.RUNTIME_GPU_TOLERATIONS)
+  /*
+   * A competition's GPU: the resource and how many units of it. The count is
+   * checked even while the resource is unset, so that a typo there does not
+   * wait to surface until the day someone turns the feature on.
+   */
+  const competitionGpuResource = parseExtendedResource('RUNTIME_COMPETITION_GPU_RESOURCE', env.RUNTIME_COMPETITION_GPU_RESOURCE)
+  const gpuCount = (env.RUNTIME_COMPETITION_GPU_COUNT ?? '').trim()
+  if (gpuCount !== '' && !/^[1-8]$/.test(gpuCount))
+    throw new Error('RUNTIME_COMPETITION_GPU_COUNT must be a whole number from 1 to 8')
+  const competitionGpuCount = gpuCount === '' ? 1 : Number(gpuCount)
+  // The rest of the GPU settings: whole numbers in their bounds, or unset.
+  const whole = (name: string, min: number, max: number, unset: number | null): number | null => {
+    const raw = (env[name] ?? '').trim()
+    if (raw === '') return unset
+    if (!/^(?:0|[1-9][0-9]{0,5})$/.test(raw) || Number(raw) < min || Number(raw) > max)
+      throw new Error(`${name} must be a whole number from ${min} to ${max}`)
+    return Number(raw)
+  }
+  const competitionGpuSlots = whole('RUNTIME_COMPETITION_GPU_SLOTS', 1, COMPETITION_GPU_SLOTS_MAX, null)
+  const competitionGpuShmMb = whole('RUNTIME_COMPETITION_GPU_SHM_MB', 0, 65536, 1024)!
+  const competitionGpuPullSeconds = whole('RUNTIME_COMPETITION_GPU_PULL_SECONDS', 0, COMPETITION_GPU_PULL_SECONDS_MAX, 900)!
   return {
     port,
     tokenFile,
@@ -201,6 +256,11 @@ export function loadRuntimeConfig(
     competitionExporterImage: env.RUNTIME_COMPETITION_EXPORTER_IMAGE ?? '',
     competitionInstanceId: env.RUNTIME_COMPETITION_INSTANCE_ID ?? '',
     competitionDataClaim: env.RUNTIME_COMPETITION_DATA_CLAIM ?? '',
+    competitionGpuResource,
+    competitionGpuCount,
+    competitionGpuSlots,
+    competitionGpuShmMb,
+    competitionGpuPullSeconds,
     // Passed through to the package proxy Pod unread (competition-jobs.ts).
     dependencyIndexUrl: (env.DEPENDENCY_INDEX_URL ?? '').trim(),
     dependencyFilesHosts: (env.DEPENDENCY_FILES_HOSTS ?? '').trim(),
diff --git a/runtime/src/http.ts b/runtime/src/http.ts
index 6466695a..0002d248 100644
--- a/runtime/src/http.ts
+++ b/runtime/src/http.ts
@@ -65,6 +65,8 @@ export function createRuntimeServer(options: Options) {
           const competition = options.jobs ? await options.jobs.health() : {
             execution: { available: false, code: 'broker_unavailable', reason: 'Competition broker is not configured' },
             preparation: { available: false, code: 'broker_unavailable', reason: 'Competition broker is not configured' },
+            gpu: { available: false, code: 'broker_unavailable', reason: 'Competition broker is not configured' },
+            gpuSlots: null,
           }
           reply(res, health.ok ? 200 : 503, { ...health, competition })
           return
diff --git a/runtime/src/index.ts b/runtime/src/index.ts
index a2a7b044..40d7f2aa 100644
--- a/runtime/src/index.ts
+++ b/runtime/src/index.ts
@@ -33,6 +33,17 @@ const jobs = new CompetitionJobs({
     priorityClass: config.priorityClass,
     podLabels: config.podLabels,
     podAnnotations: config.podAnnotations,
+    // A notebook whose competition asks for a GPU: the cluster's resource,
+    // and the GPU rooms' placement, which such a notebook shares.
+    ...(config.competitionGpuResource
+      ? { gpu: { resource: config.competitionGpuResource, count: config.competitionGpuCount,
+          slots: config.competitionGpuSlots, shmMb: config.competitionGpuShmMb } }
+      : {}),
+    gpuRuntimeClass: config.gpuRuntimeClass,
+    gpuNodeSelector: config.gpuNodeSelector,
+    gpuTolerations: config.gpuTolerations,
+    // Every Pod of a GPU environment's image, with a card or without.
+    gpuPullSeconds: config.competitionGpuPullSeconds,
   },
   kube: new HttpsKubernetesClient({
     url: config.kubeUrl,
@@ -56,7 +67,12 @@ server.listen(config.port, '0.0.0.0', () =>
   console.log(
     `Colloq runtime listening on ${config.port}; namespace ${config.namespace}` +
       `; colocation with the app ${config.colocateWithApp ? 'on' : 'off'}` +
-      `; in-place resize ${config.inPlaceResize ? 'on' : 'off'}`,
+      `; in-place resize ${config.inPlaceResize ? 'on' : 'off'}` +
+      `; competition GPU ${config.competitionGpuResource
+        ? `${config.competitionGpuResource} (${config.competitionGpuCount} per job, slots ${config.competitionGpuSlots ?? 'unset'}` +
+          `, /dev/shm ${config.competitionGpuShmMb ? `up to ${config.competitionGpuShmMb} MiB` : 'none'})`
+        : 'off'}` +
+      `; GPU image pull allowance ${config.competitionGpuPullSeconds} s`,
   ),
 )
 for (const signal of ['SIGTERM', 'SIGINT'] as const)
diff --git a/runtime/src/kubernetes.ts b/runtime/src/kubernetes.ts
index c2975d41..b54d0fe6 100644
--- a/runtime/src/kubernetes.ts
+++ b/runtime/src/kubernetes.ts
@@ -23,6 +23,44 @@ export interface KubernetesClient {
    */
   request(method: string, path: string, body?: unknown, contentType?: string): Promise
 }
+/**
+ * What the API server's "no" to a write was about, in the few words of its
+ * message we know, and nothing else of that message: the rest names the
+ * cluster's objects, its policies and other tenants' usage.
+ *
+ * - `quota`: a ResourceQuota of the namespace has no room left ("exceeded
+ *   quota"), with the quota's names for what was asked over it
+ *   (`requests.nvidia.com/gpu`, `limits.memory`), at most three;
+ * - `runtime-class`: the Pod names a RuntimeClass the cluster does not have;
+ * - `permission`: the broker's ServiceAccount may not do it (RBAC);
+ * - `admission`: something else said no at admission — a policy engine's
+ *   webhook, a ValidatingAdmissionPolicy, Pod Security, a LimitRange.
+ *
+ * A 403 is one of them (unless a node lacked a resource: `insufficient`
+ * below); a 400 or 422 only when it says an admission check denied the
+ * request (a webhook's denial is a 400 unless the webhook picks its own code),
+ * since otherwise it is a request we built wrong.
+ */
+export type KubernetesRefusal =
+  | { kind: 'quota'; resources: string[] }
+  | { kind: 'runtime-class' | 'permission' | 'admission' }
+const QUOTA = /\bexceeded quota: [^,]{0,253}, requested: (.{0,1024}?), used: /
+// The quota's resource names are sorted and joined with bare commas: `limits.memory=2Gi,requests.memory=2Gi`.
+const QUOTA_NAME = /(?:^|,)\s*([A-Za-z0-9](?:[A-Za-z0-9._/-]{0,98}[A-Za-z0-9])?)=/g
+export function refusalOf(status: number, message: string): KubernetesRefusal | undefined {
+  const text = message.slice(0, 4096)
+  if (status === 403) {
+    const quota = QUOTA.exec(text)
+    if (quota) return { kind: 'quota', resources: [...new Set([...quota[1].matchAll(QUOTA_NAME)].map((match) => match[1]))].slice(0, 3) }
+    // The RuntimeClass admission plugin: `pod rejected: RuntimeClass "nvidia" not found`.
+    if (/\bRuntimeClass "[^"]{1,253}" not found/.test(text)) return { kind: 'runtime-class' }
+    // RBAC: `User "system:serviceaccount:…" cannot create resource "pods" in API group ""…`.
+    if (/\bcannot [a-z]{1,32} resource "/.test(text)) return { kind: 'permission' }
+    return { kind: 'admission' }
+  }
+  if ((status === 400 || status === 422) && /\bdenied (?:the )?request\b/.test(text)) return { kind: 'admission' }
+  return undefined
+}
 export class KubernetesError extends Error {
   constructor(
     readonly status: number,
@@ -36,6 +74,11 @@ export class KubernetesError extends Error {
      * the operator the first; they must not be confused.
      */
     readonly insufficient?: string,
+    /**
+     * Why the API server refused it, as far as refusalOf can tell; never in
+     * the message, which stays the status and the reason word.
+     */
+    readonly refusal?: KubernetesRefusal,
   ) {
     super(
       `Kubernetes HTTP ${status}: ${/^[A-Za-z]{1,80}$/.test(reason) ? reason : 'RequestFailed'}` +
@@ -121,13 +164,15 @@ export class HttpsKubernetesClient implements KubernetesClient {
             }
             const status = res.statusCode ?? 502
             if (status < 200 || status >= 300) {
-              const lacking =
-                typeof data?.message === 'string' ? INSUFFICIENT.exec(data.message)?.[1] : undefined
+              const message = typeof data?.message === 'string' ? data.message : ''
+              const lacking = message ? INSUFFICIENT.exec(message)?.[1] : undefined
               reject(
                 new KubernetesError(
                   status,
                   typeof data?.reason === 'string' ? data.reason : 'RequestFailed',
                   lacking,
+                  // A node short of a resource is that, not an admission policy.
+                  lacking ? undefined : refusalOf(status, message),
                 ),
               )
               return
diff --git a/runtime/src/pod-policy.ts b/runtime/src/pod-policy.ts
index 9608dbe6..a3b8db73 100644
--- a/runtime/src/pod-policy.ts
+++ b/runtime/src/pod-policy.ts
@@ -202,6 +202,27 @@ export function parseRuntimeClass(raw: string | undefined): string {
   return value
 }
 
+/**
+ * RUNTIME_COMPETITION_GPU_RESOURCE: the name under which the cluster's device
+ * plugin offers a share of a card, `nvidia.com/gpu` (a whole card, or one
+ * time-sliced or MPS replica of it) or `nvidia.com/mig-1g.10gb` (a MIG slice).
+ * Empty or unset is none.
+ *
+ * Kubernetes' own rule for an extended resource, so that a typo stops the
+ * broker at start: otherwise every GPU submission would wait in the queue for
+ * a resource no node has, under the word "resources", until somebody looked.
+ * A domain-qualified name outside `kubernetes.io` (cpu and memory are not
+ * extended resources), not starting with `requests.`, which quotas reserve,
+ * and still a qualified name with that prefix, which is how a quota names it.
+ */
+export function parseExtendedResource(name: string, raw: string | undefined): string {
+  const value = (raw ?? '').trim()
+  if (!value) return ''
+  if (!value.includes('/') || value.includes('kubernetes.io/') || value.startsWith('requests.') || !qualifiedName(`requests.${value}`))
+    throw new Error(`${name} must be an extended resource name, for example nvidia.com/gpu or nvidia.com/mig-1g.10gb`)
+  return value
+}
+
 /** The label of the app's Pod that the colocation affinity looks for; the chart sets it. */
 export const APP_POD_SELECTOR = { 'colloq.dev/role': 'app' } as const
 const TOPOLOGY = 'kubernetes.io/hostname'
diff --git a/scripts/backup-local.sh b/scripts/backup-local.sh
index 14321485..a4648e59 100755
--- a/scripts/backup-local.sh
+++ b/scripts/backup-local.sh
@@ -73,7 +73,7 @@
 # second goes into the backup:
 #
 #   /kernel/environments   shipped with the product (base, base-gpu,
-#       kaggle-base).
+#       kaggle-base, kaggle-gpu).
 #       Nothing to copy: they come again with any installation of the package.
 #   /environments                created by a person. These exist
 #       nowhere else; why the directory is separate is worked through at
diff --git a/scripts/release.py b/scripts/release.py
index 76a67afc..0f488918 100755
--- a/scripts/release.py
+++ b/scripts/release.py
@@ -171,9 +171,27 @@ def read_env(file):
 # default memory and its ceiling was to edit the Deployment by hand, and the
 # next update rendered it anew and silently wiped the edit. Now they live in
 # config.env next to PUBLIC_URL and survive an update just as it does.
-RUNTIME_SETTINGS = ('RUNTIME_KERNEL_MEMORY', 'RUNTIME_KERNEL_MEMORY_MAX')
+#
+# The competition GPU pair gives the notebook of a competition that asks for a
+# GPU that many units of the named extended resource (runtime/src/config.ts);
+# on this k3s the installer's device plugin offers whole cards as
+# nvidia.com/gpu, shared with GPU rooms. With it: how many such notebooks run
+# at once (the app's queue keeps that many at the broker) and the ceiling of
+# their own /dev/shm; and for every Pod of a GPU image, how long its pull may
+# take before the job's own time starts.
+MEMORY_SETTINGS = ('RUNTIME_KERNEL_MEMORY', 'RUNTIME_KERNEL_MEMORY_MAX')
+GPU_SETTINGS = ('RUNTIME_COMPETITION_GPU_RESOURCE', 'RUNTIME_COMPETITION_GPU_COUNT',
+                'RUNTIME_COMPETITION_GPU_SLOTS', 'RUNTIME_COMPETITION_GPU_SHM_MB', 'RUNTIME_COMPETITION_GPU_PULL_SECONDS')
+# Whole numbers and their bounds, as the broker checks them (runtime/src/config.ts).
+GPU_NUMBERS = {'RUNTIME_COMPETITION_GPU_COUNT': (1, 8), 'RUNTIME_COMPETITION_GPU_SLOTS': (1, 256),
+               'RUNTIME_COMPETITION_GPU_SHM_MB': (0, 65536), 'RUNTIME_COMPETITION_GPU_PULL_SECONDS': (0, 3600)}
+RUNTIME_SETTINGS = MEMORY_SETTINGS + GPU_SETTINGS
 # The broker's default (runtime/src/config.ts): a configured ceiling is compared with it.
 RUNTIME_DEFAULT_MEMORY = '2Gi'
+# Kubernetes' rule for an extended resource name, as the broker checks it
+# (runtime/src/pod-policy.ts · parseExtendedResource): domain/name, the domain
+# outside kubernetes.io and not the quota prefix requests.
+EXTENDED_RESOURCE = re.compile(r'[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?')
 
 
 def memory_mi(value):
@@ -192,8 +210,19 @@ def runtime_settings(file):
         # An empty value means "as by default": to the broker an empty
         # environment variable is not the default but an unparseable number.
         settings = {key: val for key, val in read_env(file).items() if key in RUNTIME_SETTINGS and val}
-    for key, val in settings.items():
-        require(memory_mi(val), f'{key} must be a Kubernetes quantity in whole Mi or Gi from 64Mi to 256Gi, e.g. 4Gi')
+    for key in MEMORY_SETTINGS:
+        if key in settings:
+            require(memory_mi(settings[key]), f'{key} must be a Kubernetes quantity in whole Mi or Gi from 64Mi to 256Gi, e.g. 4Gi')
+    # The broker refuses either at start; here the update stops before anything does.
+    resource = settings.get('RUNTIME_COMPETITION_GPU_RESOURCE')
+    if resource is not None:
+        require(EXTENDED_RESOURCE.fullmatch(resource) and 'kubernetes.io/' not in resource and not resource.startswith('requests.'),
+                'RUNTIME_COMPETITION_GPU_RESOURCE must be an extended resource name, for example nvidia.com/gpu or nvidia.com/mig-1g.10gb')
+    for key, (low, high) in GPU_NUMBERS.items():
+        if key in settings:
+            value = settings[key]
+            require(re.fullmatch(r'0|[1-9][0-9]{0,5}', value) and low <= int(value) <= high,
+                    f'{key} must be a whole number from {low} to {high}')
     if 'RUNTIME_KERNEL_MEMORY_MAX' in settings:
         # Without a ceiling the broker itself takes the node's memory minus a
         # gigabyte and fits the default under it; a configured ceiling below the
diff --git a/scripts/vast-legacy.sh b/scripts/vast-legacy.sh
index f516e1b2..f4465c3c 100755
--- a/scripts/vast-legacy.sh
+++ b/scripts/vast-legacy.sh
@@ -914,8 +914,9 @@ offers = json.load(sys.stdin).get("offers") or []
 # this condition in the query not one of them is found (checked by reading,
 # twice in a row, 5 September 2026). That is, the filter threw out exactly the
 # GPUs the machine is rented for. The field in the answer is right, though —
-# the same condition, applied to the answer.
-offers = [o for o in offers if float(o.get("cuda_max_good") or 0) >= 12.1]
+# the same condition, applied to the answer. Since 13 Oct 2026 the floor is
+# 13.0: base-gpu pins torch 2.14.1 with CUDA 13.0 wheels (driver 580 or newer).
+offers = [o for o in offers if float(o.get("cuda_max_good") or 0) >= 13.0]
 for o in offers[:8]:
     print("    %-10s %s x %-10s %4.0f GB  $%.3f/hr  %s" % (
         o.get("id"), o.get("num_gpus"), o.get("gpu_name"),
diff --git a/scripts/vast-vm.sh b/scripts/vast-vm.sh
index b5ec9c7c..720ffd70 100755
--- a/scripts/vast-vm.sh
+++ b/scripts/vast-vm.sh
@@ -4,7 +4,9 @@
 #   make vast-vm                         the cheapest suitable VM offer
 #   make vast-vm GPU="RTX 3070"          only this card
 #   make vast-vm CPU=48 RAM=96           at least 48 cores and 96 GB, any card
-#   make vast-vm OFFER=51006376          exactly this offer
+#   make vast-vm GPUS=2 GPU_RAM=24       at least two cards, each with 24 GB or more
+#   make vast-vm DISK=100 MAX_PRICE=1.5  this disk (GB) and price cap ($/hr), not .env's
+#   make vast-vm OFFER=51006376          exactly this offer, however far down the market
 #   make vast-vm HOST=vsos               class address vsos. (default: the on-start's)
 #   make vast-vm INSTANCE=53175270 HOST=vsos   redeploy on a VM already rented
 #
@@ -28,8 +30,18 @@
 # the VM expect a little less than the listing: 30 of 32 cores and 49 of 63 GB
 # on 29 Sep 2026.
 #
+# A GPU class, or a competition whose submissions get a share of a card, wants
+# the cards themselves: GPUS= is how many at least (num_gpus), GPU_RAM= how
+# much memory each one has at least, in GB (gpu_ram). And disk: the CUDA
+# images are built on the VM, base-gpu about 9.9 GB and kaggle-gpu about 2.8 GB
+# more (12.7 GB in all, measured 13 Oct 2026), so DISK=100 rather than the
+# default 60.
+#
 # Reads from .env: VAST_TOKEN, VAST_SSH_KEY, VAST_IMAGE, VAST_DISK,
 # VAST_MAX_PRICE, VAST_GPU; the key is never printed or passed on a command line.
+# DISK=, MAX_PRICE= and GPU= on the command line win over theirs for one
+# rental; GPUS= and GPU_RAM= exist on the command line only, so a plain
+# `make vast-vm` searches exactly as it did before them.
 set -euo pipefail
 
 cd "$(dirname "$0")/.."
@@ -43,11 +55,20 @@ py() { python3 -c "$1"; }
 API=https://console.vast.ai/api/v0
 ONSTART="${ONSTART:-backups/vast-onstart.sh}"
 IMAGE="$(read_env VAST_IMAGE)";         IMAGE="${IMAGE:-docker.io/vastai/kvm:ubuntu_cli_22.04-2025-11-21}"
-DISK="$(read_env VAST_DISK)";           DISK="${DISK:-60}"
-MAX_PRICE="$(read_env VAST_MAX_PRICE)"; MAX_PRICE="${MAX_PRICE:-1.0}"
+DISK="${DISK:-$(read_env VAST_DISK)}";  DISK="${DISK:-60}"
+MAX_PRICE="${MAX_PRICE:-$(read_env VAST_MAX_PRICE)}"; MAX_PRICE="${MAX_PRICE:-1.0}"
 GPU_NAME="${GPU:-$(read_env VAST_GPU)}"
 MIN_CPUS="${CPU:-0}";                   MIN_RAM_GB="${RAM:-0}"
+MIN_GPUS="${GPUS:-1}";                  MIN_GPU_RAM="${GPU_RAM:-}"
 case "$MIN_CPUS$MIN_RAM_GB" in *[!0-9]*) die "CPU= and RAM= are whole numbers (cores, GB).";; esac
+case "$MIN_GPUS" in ''|*[!0-9]*|0) die "GPUS= is a whole number of cards, 1 or more.";; esac
+# They go into the query as numbers: a typo must stop here in words, not as a
+# Python traceback halfway through the search.
+number() { [[ "$1" =~ ^[0-9]+([.][0-9]+)?$ ]]; }
+number "$DISK" || die "DISK= (VAST_DISK in .env) is a size in GB, not \"$DISK\"."
+number "$MAX_PRICE" || die "MAX_PRICE= (VAST_MAX_PRICE in .env) is dollars per hour, not \"$MAX_PRICE\"."
+[ -z "$MIN_GPU_RAM" ] || number "$MIN_GPU_RAM" || die "GPU_RAM= is one card's memory in GB, not \"$MIN_GPU_RAM\"."
+case "${OFFER:-}" in *[!0-9]*) die "OFFER= is an offer number from the list, not \"$OFFER\".";; esac
 SSH_KEY="$(read_env VAST_SSH_KEY)";     SSH_KEY="${SSH_KEY:-$HOME/.ssh/id_ed25519}"
 # shellcheck disable=SC2088
 case "$SSH_KEY" in "~/"*) SSH_KEY="$HOME/${SSH_KEY#\~/}" ;; esac
@@ -106,23 +127,40 @@ if [ -n "${INSTANCE:-}" ]; then
   say "${BOLD}instance $id${OFF}: deploying ${ADDRESS:+for $ADDRESS }without renting"
 else
 # ---------------------------------------------------------------- offer
-query="$(DISK="$DISK" MAX_PRICE="$MAX_PRICE" GPU_NAME="$GPU_NAME" MIN_CPUS="$MIN_CPUS" MIN_RAM_GB="$MIN_RAM_GB" py '
+query="$(DISK="$DISK" MAX_PRICE="$MAX_PRICE" GPU_NAME="$GPU_NAME" MIN_CPUS="$MIN_CPUS" MIN_RAM_GB="$MIN_RAM_GB" \
+  MIN_GPUS="$MIN_GPUS" MIN_GPU_RAM="$MIN_GPU_RAM" OFFER="${OFFER:-}" py '
 import json, os
 q = {"vms_enabled": {"eq": True}, "type": "ondemand", "rentable": {"eq": True}, "rented": {"eq": False},
-     "verified": {"eq": True}, "reliability": {"gte": 0.98}, "num_gpus": {"gte": 1},
+     "verified": {"eq": True}, "reliability": {"gte": 0.98}, "num_gpus": {"gte": int(os.environ["MIN_GPUS"])},
      "disk_space": {"gte": float(os.environ["DISK"]) + 10},
      "dph_total": {"lte": float(os.environ["MAX_PRICE"])}, "order": [["dph_total", "asc"]], "limit": 60}
 if os.environ["GPU_NAME"].strip(): q["gpu_name"] = {"in": [os.environ["GPU_NAME"].strip()]}
 if int(os.environ["MIN_CPUS"]): q["cpu_cores_effective"] = {"gte": int(os.environ["MIN_CPUS"])}
 if int(os.environ["MIN_RAM_GB"]): q["cpu_ram"] = {"gte": int(os.environ["MIN_RAM_GB"]) * 1024}
+# Megabytes, and deliberately not GB * 1024: a "24 GB" card reports 24564 MB,
+# and a round power of two would cut off every 4090 (as in vast-legacy.sh).
+if os.environ["MIN_GPU_RAM"]: q["gpu_ram"] = {"gte": int(float(os.environ["MIN_GPU_RAM"]) * 1000)}
+# OFFER is asked for by number, not by "id": on 13 Oct 2026 an "id" condition
+# made the bundles API answer with no offers at all (an earlier note here said
+# it was ignored), and no answer holds more than 64 offers, whatever "limit"
+# says. Matched only in the answer, as it was here before, OFFER was found
+# among the 60 cheapest and nowhere else: a better card or a known host further
+# down the market was "not found" although it passed every filter.
+# ask_contract_id holds the same number, and the API keeps to it. The other
+# conditions stay, so OFFER still cannot rent what the price cap, the disk or
+# the VM filters refuse.
+if os.environ["OFFER"]: q["ask_contract_id"] = {"eq": int(os.environ["OFFER"])}
 print(json.dumps(q))
 ')"
 # cuda_max_good is checked on the answer, not in the query: the server-side
-# filter hid RTX 5070/5090 offers in September 2026 (see vast-legacy.sh).
-# The bundles API silently ignores an "id" condition, so OFFER is matched here.
+# filter hid RTX 5070/5090 offers in September 2026 (see vast-legacy.sh). The
+# floor is 13.0 since 13 Oct 2026: kernel/environments/base-gpu.txt pins torch
+# 2.14.1 with CUDA 13.0 wheels, which a host with a CUDA 12 driver cannot run.
+# OFFER is matched in the answer as well, in case the API ever drops the
+# condition above as quietly as it once dropped "id".
 suitable="$(api POST bundles/ "$query" | OFFER="${OFFER:-}" py '
 import json, os, sys
-offers = [o for o in json.load(sys.stdin).get("offers") or [] if float(o.get("cuda_max_good") or 0) >= 12.1]
+offers = [o for o in json.load(sys.stdin).get("offers") or [] if float(o.get("cuda_max_good") or 0) >= 13.0]
 if os.environ["OFFER"].strip(): offers = [o for o in offers if str(o["id"]) == os.environ["OFFER"].strip()]
 for o in offers[:8]:
     print("    %-10s %s x %-12s %3.0f GB  %3.0f cores  %4.0f GB RAM  $%.3f/hr  %s" % (o["id"], o.get("num_gpus"),
@@ -130,7 +168,9 @@ for o in offers[:8]:
           (o.get("cpu_ram") or 0) / 1024, o.get("dph_total") or 0, o.get("geolocation") or ""), file=sys.stderr)
 for o in offers: print("%s\t%.3f\t%s x %s" % (o["id"], o.get("dph_total") or 0, o.get("num_gpus"), o.get("gpu_name")))
 ')"
-[ -n "$suitable" ] || die "no VM offer matches: GPU \"${GPU_NAME:-any}\"${OFFER:+, offer $OFFER}${CPU:+, from $CPU cores}${RAM:+, from $RAM GB RAM}, up to \$$MAX_PRICE/hr, disk from $DISK GB."
+# The Max CUDA floor above is named with the other conditions: an offer it
+# alone refuses passes every one of those, and a message without it blamed none.
+[ -n "$suitable" ] || die "no VM offer matches: GPU \"${GPU_NAME:-any}\"${OFFER:+, offer $OFFER}${GPUS:+, from $GPUS cards}${GPU_RAM:+, from $GPU_RAM GB per card}${CPU:+, from $CPU cores}${RAM:+, from $RAM GB RAM}, up to \$$MAX_PRICE/hr, disk from $DISK GB, Max CUDA from 13.0."
 IFS=$'\t' read -r offer_id price what <<<"$(printf '%s\n' "$suitable" | head -n 1)"
 say ""
 say "${BOLD}VM offer $offer_id${OFF}: $what, \$$price/hr, image $(sed -n 's/^export COLLOQ_IMAGE="\(.*\)"/\1/p' "$SCRIPT")${ADDRESS:+, address $ADDRESS}"
diff --git a/scripts/vast.sh b/scripts/vast.sh
index 9e2bbc15..50e33abc 100755
--- a/scripts/vast.sh
+++ b/scripts/vast.sh
@@ -917,8 +917,9 @@ offers = json.load(sys.stdin).get("offers") or []
 # this condition in the query not one of them is found (checked by reading,
 # twice in a row, 5 September 2026). That is, the filter threw out exactly the
 # GPUs the machine is rented for. The field in the answer is right, though —
-# the same condition, applied to the answer.
-offers = [o for o in offers if float(o.get("cuda_max_good") or 0) >= 12.1]
+# the same condition, applied to the answer. Since 13 Oct 2026 the floor is
+# 13.0: base-gpu pins torch 2.14.1 with CUDA 13.0 wheels (driver 580 or newer).
+offers = [o for o in offers if float(o.get("cuda_max_good") or 0) >= 13.0]
 for o in offers[:8]:
     print("    %-10s %s x %-10s %4.0f GB  $%.3f/hr  %s" % (
         o.get("id"), o.get("num_gpus"), o.get("gpu_name"),
diff --git a/server/src/competitions/broker-runner.ts b/server/src/competitions/broker-runner.ts
index 728898c9..9450863a 100644
--- a/server/src/competitions/broker-runner.ts
+++ b/server/src/competitions/broker-runner.ts
@@ -2,7 +2,7 @@ import { randomUUID } from 'node:crypto'
 import path from 'node:path'
 import { imageRevision, kernelRuntimeClient, loadRuntimeCatalog, type RuntimeClient } from '../kernel/runtime-client.js'
 import { resolveRuntimeEnvironment } from '@shared/runtime'
-import { parseCompetitionJobIntent, type CompetitionJobIntent, type CompetitionJobStatus } from '@shared/competition-runtime'
+import { COMPETITION_GPU_PULL_SECONDS_MAX, COMPETITION_RESOURCES_UNAVAILABLE, parseCompetitionJobIntent, type CompetitionJobIntent, type CompetitionJobStatus } from '@shared/competition-runtime'
 import { competitionsFs, attemptDir, attemptPath } from './storage.js'
 import { DEFAULT_ID_COLUMN, dependencyFailureOf, verdictOfRun } from './docker-runner.js'
 import { brokerHarnessDir } from './harness.js'
@@ -24,12 +24,28 @@ function report(file:string):Record|null {
   } catch { return null }
 }
 const terminal = (phase:CompetitionJobStatus['phase']) => ['complete','failed','cancelled'].includes(phase)
-export interface BrokerRunnerOptions { catalog?: typeof loadRuntimeCatalog; pollMs?: number }
+/** `now`: the clock the job deadlines are read on, for tests. */
+export interface BrokerRunnerOptions { catalog?: typeof loadRuntimeCatalog; pollMs?: number; now?: () => number }
+/**
+ * `gpu`: no node had a free unit of the cluster's GPU resource for a notebook
+ * given a card, or the API server refused its Pod (a quota, a RuntimeClass, an
+ * admission policy).
+ *
+ * `detail`: the broker's own words for why, when it has more to say than that
+ * resources are unavailable (shared/competition-runtime.ts ·
+ * COMPETITION_RESOURCES_UNAVAILABLE) — the teacher reads them while the job
+ * waits in the queue, so that a quota used up or a RuntimeClass missing on the
+ * cluster is not a mystery behind "unschedulable".
+ */
 export class CompetitionResourcePending extends Error {
-  constructor(readonly resource:'memory'|'cpu'|'storage'|'other'='other',readonly kind:'notebook'|'metric'='notebook') {
+  constructor(readonly resource:'memory'|'cpu'|'storage'|'gpu'|'other'='other',readonly kind:'notebook'|'metric'='notebook',readonly detail:string|null=null) {
     super('Competition resources are unavailable; the job will retry from the queue.')
     this.name='CompetitionResourcePending'
   }
+  /** What the teacher reads while the job waits (runner.ts · runJob). */
+  get teacherReason():string {
+    return `Competition Pod unschedulable (${this.resource})${this.detail?`: ${this.detail}`:''}; retrying automatically.`
+  }
 }
 
 /** The broker alone maps these typed IDs to PVC subPaths and owns Pods. */
@@ -38,33 +54,56 @@ export class BrokerCompetitionRunner implements CompetitionRunner {
   private readonly jobs=new Map()
   constructor(private readonly client:Pick = kernelRuntimeClient(), private readonly options:BrokerRunnerOptions={}) {}
 
-  private revision(environment:string, image:string|undefined):string {
+  /** The catalog revision of a pinned image, and whether its environment is a GPU one. */
+  private environment(environment:string, image:string|undefined):{revision:string;gpu:boolean} {
     if (!image || !image.includes('@')) throw new Error('Competition broker requires a pinned catalog image')
     const revision=imageRevision(image)
     const selected=resolveRuntimeEnvironment((this.options.catalog??loadRuntimeCatalog)(),environment,revision)
     if (selected.image!==image) throw new Error('Competition image is not in the runtime catalog')
-    return revision
+    return {revision,gpu:selected.gpu}
   }
-  private async execute(container:string,intent:CompetitionJobIntent,signal?:AbortSignal,onProgress?:(status:CompetitionJobStatus)=>void):Promise {
+  private async execute(container:string,intent:CompetitionJobIntent,gpuImage:boolean,signal?:AbortSignal,onProgress?:(status:CompetitionJobStatus)=>void):Promise {
     const job=parseCompetitionJobIntent(intent)
+    const now=this.options.now??Date.now
     if (signal?.aborted) throw signal.reason ?? new Error('Competition job cancelled')
     this.jobs.set(container,job.jobId)
     const abort=()=>{void this.client.cancelCompetitionJob(job.jobId).catch(()=>undefined)}
     signal?.addEventListener('abort',abort,{once:true})
     try {
       let status=await this.client.startCompetitionJob(job)
-      const deadline=Date.now()+job.limits.wallSeconds*1000+60_000
+      /*
+       * How long the job may take before the broker is given up on: its limit
+       * and a minute to export, from the moment the start came back.
+       *
+       * A GPU environment's image (10 to 13 GB) may first be pulled onto a
+       * node that never ran it, and the broker gives it time for that before
+       * the job's own clock starts (RUNTIME_COMPETITION_GPU_PULL_SECONDS,
+       * competition-jobs.ts · pullSeconds). Counted from the start call, the
+       * pull came out of the notebook's own time here too, and a notebook
+       * that trained to its limit was cancelled as late. So for such an image
+       * the limit counts from the moment `main` is seen started, as the
+       * broker's and the harness's do (run_notebook.py · stop_at), and while
+       * it is still queued the most any broker may allow for the pull
+       * (COMPETITION_GPU_PULL_SECONDS_MAX) and its 30 seconds of startup come
+       * on top: the broker says when a pull took too long, not this loop. A
+       * CPU image keeps the one deadline it always had.
+       */
+      const limit=job.limits.wallSeconds*1000+60_000
+      let deadline=now()+limit+(gpuImage?(30+COMPETITION_GPU_PULL_SECONDS_MAX)*1000:0)
+      let started=!gpuImage
       let collected=false
       while(!terminal(status.phase)) {
         if (signal?.aborted) throw signal.reason ?? new Error('Competition job cancelled')
-        if (Date.now()>deadline) { await this.client.cancelCompetitionJob(job.jobId); throw new Error('Competition job exceeded its deadline') }
+        if (!started&&status.phase!=='queued') { started=true;deadline=now()+limit }
+        if (now()>deadline) { await this.client.cancelCompetitionJob(job.jobId); throw new Error('Competition job exceeded its deadline') }
         if(status.phase==='exporting') { await this.client.collectCompetitionJob(job.jobId);collected=true;status=await this.client.competitionJob(job.jobId);continue }
         onProgress?.(status)
         await sleep(this.options.pollMs??500)
         status=await this.client.competitionJob(job.jobId)
       }
       if (signal?.aborted) throw signal.reason ?? new Error('Competition job cancelled')
-      if(status.failure?.code==='unschedulable')throw new CompetitionResourcePending(status.failure.resource??'other',job.kind==='metric'?'metric':'notebook')
+      if(status.failure?.code==='unschedulable')throw new CompetitionResourcePending(status.failure.resource??'other',job.kind==='metric'?'metric':'notebook',
+        status.error&&status.error!==COMPETITION_RESOURCES_UNAVAILABLE?status.error:null)
       if (status.phase==='complete'&&!collected) await this.client.collectCompetitionJob(job.jobId)
       return status
     } finally {
@@ -75,7 +114,7 @@ export class BrokerCompetitionRunner implements CompetitionRunner {
   async run(request:RunRequest):Promise {
     const started=Date.now()
     if (!request.attemptId) throw new Error('Competition broker requires an attempt ID')
-    const revision=this.revision(request.competition.environment,request.imageDigest)
+    const {revision,gpu:gpuImage}=this.environment(request.competition.environment,request.imageDigest)
     brokerHarnessDir()
     competitionsFs.mkdirSync(request.resultDir,{recursive:true})
     competitionsFs.chmodSync(request.resultDir,0o700)
@@ -89,13 +128,21 @@ export class BrokerCompetitionRunner implements CompetitionRunner {
       throw new Error('Competition broker: sealed inputs must be the attempt inputs folder')
     }
     const jobId=randomUUID().replaceAll('-','')
+    /*
+     * A card only when the queue gave this attempt one (runner-port.ts ·
+     * RunRequest.gpu), and then as a bare flag: which resource and how many
+     * units are the cluster's to say (RUNTIME_COMPETITION_GPU_*). Never for a
+     * notebook without one: a broker older than GPU jobs refuses the unknown
+     * key, and every CPU submission would fail with it.
+     */
     const intent:CompetitionJobIntent={schemaVersion:1,jobId,kind:'notebook',environment:request.competition.environment,revision,
       competitionId:request.competition.id,submissionId:request.submissionId,attemptId:request.attemptId,...(bundleId?{bundleId}:{}),
       ...(request.sealedInputs?{sealedInputs:true as const}:{}),
+      ...(request.gpu?{gpu:true as const}:{}),
       limits:{wallSeconds:request.limits.wallSeconds,memoryMb:request.limits.memoryMb,cpus:request.limits.cpus,pids:request.limits.pids,tmpfsMb:request.limits.tmpfsMb,targetBytes:request.limits.targetBytes,
         ...(packagesMb?{packagesMb}:{})}}
     let progress:CompetitionJobStatus['progress']=null
-    const status=await this.execute(request.container,intent,request.signal,s=>{
+    const status=await this.execute(request.container,intent,gpuImage,request.signal,s=>{
       if (!s.progress || s.progress.phase==='score') return
       const next=s.progress
       if (JSON.stringify(progress)===JSON.stringify(next)) return
@@ -119,17 +166,18 @@ export class BrokerCompetitionRunner implements CompetitionRunner {
   async score(request:ScoreRequest):Promise {
     const started=Date.now()
     if (!request.attemptId) throw new Error('Competition broker requires an attempt ID')
-    const revision=this.revision(request.competition.environment,request.imageDigest)
+    const {revision,gpu:gpuImage}=this.environment(request.competition.environment,request.imageDigest)
     brokerHarnessDir()
     competitionsFs.mkdirSync(request.outDir,{recursive:true})
     competitionsFs.chmodSync(request.outDir,0o700)
     const config=attemptDir(request.competition.id,request.submissionId,request.attemptId,'score-config')
     competitionsFs.chmodSync(config,0o700)
     competitionsFs.writeFileSync(path.join(config,'request.json'),JSON.stringify({idColumn:DEFAULT_ID_COLUMN,publicPercent:request.competition.publicPercent,splitSeed:request.competition.splitSeed}),{mode:0o600})
+    // Never a card for the metric, whatever the notebook had: it holds the hidden answers.
     const intent:CompetitionJobIntent={schemaVersion:1,jobId:randomUUID().replaceAll('-',''),kind:'metric',environment:request.competition.environment,revision,
       competitionId:request.competition.id,submissionId:request.submissionId,attemptId:request.attemptId,
       limits:{wallSeconds:request.limits.wallSeconds,memoryMb:request.limits.memoryMb,cpus:request.limits.cpus,pids:request.limits.pids,tmpfsMb:request.limits.tmpfsMb,targetBytes:request.limits.targetBytes}}
-    const status=await this.execute(request.container,intent,request.signal)
+    const status=await this.execute(request.container,intent,gpuImage,request.signal)
     const raw=report(path.join(request.outDir,'score.json'))
     const score=raw?.attemptId===request.attemptId?raw:null
     const valid=!!score && reportAllowed(status,score.status)
diff --git a/server/src/competitions/capabilities.ts b/server/src/competitions/capabilities.ts
index 7fba29da..d2863f3e 100644
--- a/server/src/competitions/capabilities.ts
+++ b/server/src/competitions/capabilities.ts
@@ -5,6 +5,7 @@ import { config } from '../config.js'
 import { imageRevision, kernelRuntimeClient, loadRuntimeCatalog } from '../kernel/runtime-client.js'
 import { resolveRuntimeEnvironment } from '@shared/runtime'
 import { competitionBackend } from './runner-port.js'
+import { runnerGpuDevices } from './gpu.js'
 import { tr } from '@shared/i18n'
 import type { CompetitionCapabilities, RuntimeCapability } from '@shared/capabilities'
 
@@ -17,7 +18,25 @@ function dockerMajor(version: string | null | undefined): number | null {
   const major = matched ? Number(matched[1]) : NaN
   return Number.isSafeInteger(major) ? major : null
 }
-export function evaluateCapabilities(input: { backend: Backend; dockerVersion?: string | null; imageAvailable?: boolean; storageAvailable?: boolean; brokerCapabilities?: CompetitionCapabilities }): CompetitionCapabilities {
+/**
+ * `gpu` — the competition wants a card (CompetitionLimits.gpu); `runnerGpus`
+ * — the owner gave this machine's submissions at least one (docker and the
+ * test stand-in). Under the broker the second answer is the broker's own
+ * `gpu` capability, and a broker that does not report one has none.
+ *
+ * `gpu_unavailable` is an EXECUTION verdict and comes last: a dead docker or a
+ * missing image is said first, being the more basic trouble. Preparation is
+ * judged before it and never inherits it: installing a participant's packages
+ * needs no card. Without a card a GPU competition is refused at the door —
+ * the upload and the baseline check — instead of waiting in the queue for a
+ * card that nobody gave, or failing as a metric on the broker.
+ *
+ * Under the broker the answer also carries the broker's `gpuSlots` as it
+ * came (absent when it said none): how many GPU notebooks the cluster has
+ * units for at once, which the pump keeps its GPU work to (runner.ts ·
+ * queueCards). Not a verdict, and never a reason to refuse anything here.
+ */
+export function evaluateCapabilities(input: { backend: Backend; dockerVersion?: string | null; imageAvailable?: boolean; storageAvailable?: boolean; brokerCapabilities?: CompetitionCapabilities; gpu?: boolean; runnerGpus?: boolean }): CompetitionCapabilities {
   let execution = ok()
   if (input.backend === 'broker') execution = input.brokerCapabilities?.execution ?? unavailable('broker_unavailable', 'runtime.brokerUnavailable')
   else if (input.backend === 'docker' && !input.dockerVersion) execution = unavailable('docker_unavailable', 'runtime.dockerUnavailable')
@@ -31,7 +50,26 @@ export function evaluateCapabilities(input: { backend: Backend; dockerVersion?:
     if(preparation.available&&input.storageAvailable===false)preparation=unavailable('storage_unavailable','runtime.storageUnavailable')
   }else preparation=!execution.available?{...execution}:input.backend==='docker'&&(major===null||major<28)
     ?unavailable('docker_version','runtime.preparationVersion'):ok()
-  return { execution, preparation }
+  if (execution.available && input.gpu === true) {
+    // Read defensively: `gpu` is a newer field of the broker's health.
+    const cards = input.backend === 'broker' ? input.brokerCapabilities?.gpu?.available === true : input.runnerGpus === true
+    if (!cards) execution = unavailable('gpu_unavailable', input.backend === 'broker' ? 'runtime.gpuUnavailableCluster' : 'runtime.gpuUnavailable')
+  }
+  const gpuSlots = input.backend === 'broker' ? input.brokerCapabilities?.gpuSlots : undefined
+  return { execution, preparation, ...(gpuSlots !== undefined ? { gpuSlots } : {}) }
+}
+
+/** What a check of ONE competition adds to the image: whether its notebook wants a card. */
+export interface CompetitionNeeds { gpu?: boolean }
+
+/**
+ * What a door that runs a competition's NOTEBOOK asks of the executor: its
+ * card, when the competition has one. Only those doors: the metric never gets
+ * a card, and a door that runs only the metric (a rescore, the metric check)
+ * or no notebook at all (package preparation, a draft) asks for none.
+ */
+export function notebookNeeds(competition: { limits: { gpu?: boolean } }): CompetitionNeeds {
+  return { gpu: competition.limits.gpu === true }
 }
 
 type Command = (args: string[]) => Promise
@@ -43,12 +81,22 @@ interface CapabilityReaderOptions {
   storageAvailable?: () => boolean
   brokerCapabilities?: () => Promise
   catalogAvailable?: (environmentName?:string,imageDigest?:string)=>boolean
+  /** Whether any card is given to submissions here (docker and the test stand-in). */
+  runnerGpus?: () => boolean
 }
 /** One short cache for both successful and failed probes. In-flight callers
- * share the promise; expiry starts at completion, so slow probes also coalesce. */
+ * share the promise; expiry starts at completion, so slow probes also coalesce.
+ *
+ * The cache holds what is slow to ask — the daemon, an image, the broker —
+ * and never a verdict: whether the competition wants a card is applied to
+ * every answer afresh, and whether the machine has runner cards is read at
+ * each call (a settings row and KERNEL_GPUS, no process). So a GPU
+ * competition and a CPU one on the same image never share a verdict, and a
+ * card the owner just gave is seen at the next request. */
 export function createCapabilityReader(options: CapabilityReaderOptions = {}) {
   const clock = options.clock ?? Date.now
   const backendOf = options.backend ?? (() => competitionBackend())
+  const runnerGpus = options.runnerGpus ?? (() => runnerGpuDevices().length > 0)
   const run = options.run ?? (async (args) => (await execute('docker', args, { timeout: 3000, maxBuffer: 4096 })).stdout.trim())
   const storageAvailable = options.storageAvailable ?? (() => {
     try { fs.mkdirSync(config.dataDir, { recursive: true }); fs.accessSync(config.dataDir, fs.constants.R_OK | fs.constants.W_OK); return true }
@@ -73,15 +121,16 @@ export function createCapabilityReader(options: CapabilityReaderOptions = {}) {
     entry.value.finally(() => { entry.until = clock() + 5000 }).catch(() => undefined)
     return entry
   }
-  return async (environmentName?: string, imageDigest?: string): Promise => {
+  return async (environmentName?: string, imageDigest?: string, needs: CompetitionNeeds = {}): Promise => {
     const backend = backendOf()
+    const gpu = needs.gpu === true
     if(backend==='broker') {
       if(!broker||broker.until<=clock())broker=probe(()=>brokerCapabilities().catch(()=>null))
       let imageAvailable=false
       try{imageAvailable=catalogAvailable(environmentName,imageDigest)}catch{ /* catalog unavailable */ }
-      return evaluateCapabilities({backend,brokerCapabilities:await broker.value??undefined,imageAvailable,storageAvailable:storageAvailable()})
+      return evaluateCapabilities({backend,brokerCapabilities:await broker.value??undefined,imageAvailable,storageAvailable:storageAvailable(),gpu})
     }
-    if (backend !== 'docker') return evaluateCapabilities({ backend })
+    if (backend !== 'docker') return evaluateCapabilities({ backend, gpu, runnerGpus: gpu && runnerGpus() })
     if (!daemon || daemon.until <= clock()) daemon = probe(() => run(['version', '--format', '{{.Server.Version}}']).then(value => value || null, () => null))
     const version = await daemon.value
     if (!version) return evaluateCapabilities({ backend, dockerVersion: null })
@@ -96,11 +145,18 @@ export function createCapabilityReader(options: CapabilityReaderOptions = {}) {
       }
       imageAvailable = await cached.value
     }
-    return evaluateCapabilities({ backend, dockerVersion: version, imageAvailable, storageAvailable: storageAvailable() })
+    return evaluateCapabilities({ backend, dockerVersion: version, imageAvailable, storageAvailable: storageAvailable(), gpu, runnerGpus: gpu && runnerGpus() })
   }
 }
 let reader = createCapabilityReader()
-export const competitionCapabilities = (environmentName?: string, imageDigest?: string): Promise => reader(environmentName, imageDigest)
+/**
+ * What the executor can do now; with an environment (and its pinned image)
+ * for one competition, and `needs.gpu` when that competition's notebook
+ * wants a card (`{ gpu: competition.limits.gpu === true }`). The instance-wide
+ * question — the pump, the health page — passes nothing and is never refused
+ * for a card.
+ */
+export const competitionCapabilities = (environmentName?: string, imageDigest?: string, needs: CompetitionNeeds = {}): Promise => reader(environmentName, imageDigest, needs)
 /**
  * Swap the probe out — for tests that need a runtime answering slowly: the
  * test backend answers in the same tick, and what needs checking is what
@@ -113,7 +169,7 @@ export class RuntimeUnavailableError extends Error {
   readonly status = 503
   constructor(public readonly code: string, message: string) { super(message); this.name = 'RuntimeUnavailableError' }
 }
-export async function assertCompetitionCapability(kind: 'execution' | 'preparation', environmentName?: string, imageDigest?: string): Promise {
-  const capability = (await competitionCapabilities(environmentName, imageDigest))[kind]
+export async function assertCompetitionCapability(kind: 'execution' | 'preparation', environmentName?: string, imageDigest?: string, needs: CompetitionNeeds = {}): Promise {
+  const capability = (await competitionCapabilities(environmentName, imageDigest, needs))[kind]
   if (!capability.available) throw new RuntimeUnavailableError(capability.code, capability.reason ?? tr('runtime.prepareUnavailable'))
 }
diff --git a/server/src/competitions/docker-runner.ts b/server/src/competitions/docker-runner.ts
index 1c4223c8..2dda1f3b 100644
--- a/server/src/competitions/docker-runner.ts
+++ b/server/src/competitions/docker-runner.ts
@@ -50,6 +50,21 @@
  *    honest, unlike in a room, where it sticks forever and postmortem.ts has
  *    to count kills with a cgroup counter: the container lives once.
  *
+ * 8. A video card comes back only for a notebook the queue gave one
+ *    (RunRequest.gpu, competitions/gpu.ts), with what a room gets for it:
+ *    `--gpus device=`, a /dev/shm big enough for DataLoader workers (docker's
+ *    64 MB kills them with a bus error; it counts toward `--memory`), and the
+ *    `colloq.gpu` label, by which a class is kept off the card while the run
+ *    holds it (kernel/pool.ts · submissionGpuCards). Every hardening flag
+ *    stays: the card was tried under `--read-only`, `--cap-drop=ALL`, uid
+ *    1000 and `--network none` on the lab VM. A card shared by several runs
+ *    has each held to its share twice: torch's allocator and CatBoost's GPU
+ *    pool are capped from inside (harness.ts · SITECUSTOMIZE), and the
+ *    container is watched from outside (VramWatch), because the inside is
+ *    the participant's own. The watch is not theirs either: it samples as
+ *    root, whom their uid cannot signal, and it fails closed — a run on a
+ *    shared card that cannot be read is stopped, not left unwatched.
+ *
  * And one rule, bought at a higher price than any other: we mount ONLY
  * directories, and only at paths that did not exist before this submission
  * (see the header of storage.ts).
@@ -59,10 +74,12 @@ import path from 'node:path'
 import { createHash, randomUUID } from 'node:crypto'
 import { Transform, type TransformCallback, type Writable } from 'node:stream'
 import { pipeline } from 'node:stream/promises'
+import { tr } from '@shared/i18n'
 import { config } from '../config.js'
 import { competitionsFs, hostPathOf, EXECUTED_BYTES, EXECUTED_FILE } from './storage.js'
-import { harnessDir } from './harness.js'
+import { HARNESS_SITE, harnessDir } from './harness.js'
 import { competitionMemory } from './capacity.js'
+import { gpuAllocatorMb, gpuKillAboveMb } from './gpu.js'
 import {
   METRIC_WALL_SECONDS,
   registerCompetitionRunner,
@@ -203,22 +220,33 @@ async function retried(
 }
 
 /**
- * Containers whose removal failed: name → the memory they may still hold.
+ * What a container still holds while its removal is pending: memory, and the
+ * runner card it was given (`null` — none).
+ */
+interface Held {
+  memoryMb: number
+  gpu: string | null
+}
+
+/**
+ * Containers whose removal failed: name → what they may still hold.
  *
  * A failed `docker rm` after a finished run is the daemon's trouble, not the
  * submission's: the verdict stands, the removal is retried later, and until
  * it succeeds only THAT container's memory stays out of the queue's reach.
- * It used to close the whole queue.
+ * It used to close the whole queue. Its card too: a notebook that may still
+ * be on the card holds its share, and the next GPU run goes to it only after
+ * the container is gone (capacity · heldGpus).
  */
-const pendingCleanup = new Map()
+const pendingCleanup = new Map()
 let cleanupTimer: NodeJS.Timeout | null = null
 const CLEANUP_RETRY_MS = 30_000
 let cleanupFailures = 0
 let unverifiedLegacyContainers = 0
 export function competitionDockerDiagnostics() { return { cleanupFailures, pendingCleanup: pendingCleanup.size, unverifiedLegacyContainers } }
 
-/** Remove a container; `false` means it is left for the retry, holding `memoryMb`. Never throws. */
-async function removeContainer(container: string, memoryMb = 0, patient = true): Promise {
+/** Remove a container; `false` means it is left for the retry, holding what `held` says. Never throws. */
+async function removeContainer(container: string, held: Partial = {}, patient = true): Promise {
   const removal = () => docker(['rm', '-f', container], 60_000)
   const gone = (result: { code: number; out: string }) => result.code === 0 || missing(result.out)
   const result = patient ? await retried(removal, gone) : await removal()
@@ -230,7 +258,11 @@ async function removeContainer(container: string, memoryMb = 0, patient = true):
     cleanupFailures = Math.min(Number.MAX_SAFE_INTEGER, cleanupFailures + 1)
     console.warn('[competitions] container removal failed; retrying later', container)
   }
-  pendingCleanup.set(container, Math.max(pendingCleanup.get(container) ?? 0, memoryMb))
+  const before = pendingCleanup.get(container)
+  pendingCleanup.set(container, {
+    memoryMb: Math.max(before?.memoryMb ?? 0, held.memoryMb ?? 0),
+    gpu: held.gpu || before?.gpu || null,
+  })
   scheduleCleanup()
   return false
 }
@@ -241,7 +273,7 @@ let retrying: Promise | null = null
 /** One more try for every container still awaiting removal; the next pump or timer is the backoff. */
 function retryPendingCleanup(): Promise {
   retrying ??= (async () => {
-    for (const [container, memoryMb] of [...pendingCleanup]) await removeContainer(container, memoryMb, false)
+    for (const [container, held] of [...pendingCleanup]) await removeContainer(container, held, false)
   })().finally(() => { retrying = null })
   return retrying
 }
@@ -338,6 +370,39 @@ function mount(inside: string, at: string, readOnly = false): string[] {
   return ['-v', `${hostPathOf(inside)}:${at}${readOnly ? ':ro' : ''}`]
 }
 
+/**
+ * The video memory a run is held to on its card, MB, or `null` — nothing to
+ * hold: no card, a whole card to itself (`perDevice` 1, nobody to disturb),
+ * or a card that did not say its size (a limit nobody can state is not one
+ * to enforce). Both halves of the share key off this one answer: the cap
+ * inside (runArgs) and the watchdog outside (VramWatch).
+ */
+export function heldShareMb(gpu: RunRequest['gpu']): number | null {
+  if (!gpu?.device || gpu.sliceMb === null || !(gpu.perDevice > 1)) return null
+  return gpu.sliceMb
+}
+
+/**
+ * /dev/shm for a run with a card, MB: half its memory, at most a gigabyte —
+ * what a room gets by default (KERNEL_SHM). It is memory-backed and counted
+ * in `--memory`, so a bigger one is only a bigger way to die of memory.
+ */
+function shmMb(memoryMb: number): number {
+  return Math.min(1024, Math.floor(memoryMb / 2))
+}
+
+/**
+ * PYTHONPATH from its entries, in order; nothing at all without any.
+ *
+ * The one variable both the runner and the image could want: the kernel
+ * images set none (kernel/Dockerfile), so the entries here are the whole of
+ * it, and a second one the runner adds later goes into this list rather than
+ * into a second `-e` that would replace the first.
+ */
+function pythonPath(entries: readonly string[]): string[] {
+  return entries.length > 0 ? ['-e', `PYTHONPATH=${entries.join(':')}`] : []
+}
+
 /**
  * `docker run` arguments for the NOTEBOOK container.
  *
@@ -345,6 +410,8 @@ function mount(inside: string, at: string, readOnly = false): string[] {
  * assembling this line correctly matters more than calling it: a lost
  * `--network none` here is a submission with internet access, and it can only
  * be noticed when someone makes use of it.
+ *
+ * Without `gpu` the line is exactly what it was before cards existed.
  */
 export function runArgs(opts: {
   attemptId?: string
@@ -358,12 +425,16 @@ export function runArgs(opts: {
   packagesMb?: number
   limits: StepLimits
   target: string
+  /** The card the queue gave this run (RunRequest.gpu); an empty device is none of ours. */
+  gpu?: RunRequest['gpu']
 }): string[] {
   const { container, limits } = opts
   const memory = `${limits.memoryMb}m`
   const packagesMb = opts.dependenciesDir && opts.packagesMb ? Math.max(1, Math.ceil(opts.packagesMb)) : 0
   // No file of the set is bigger than its room, and every file lives in a capped tmpfs.
   const fsize = Math.max(limits.fsizeBytes, packagesMb * 1024 ** 2)
+  const device = opts.gpu?.device || null
+  const share = heldShareMb(opts.gpu)
   return [
     'run',
     '-d',
@@ -427,6 +498,20 @@ export function runArgs(opts: {
     `--tmpfs=/result:rw,nosuid,nodev,size=${Math.max(1, Math.ceil(limits.targetBytes * 2 / 1024 ** 2))}m,mode=1777`,
     '-e', `COMP_ATTEMPT_ID=${opts.attemptId ?? opts.container}`,
     '-e', `COMP_MAX_OUTPUT_KILL_BYTES=${limits.outputKillBytes}`,
+    /*
+     * A share of a card: torch's allocator is capped a little below it (the
+     * CUDA context lives outside the allocator, competitions/gpu.ts ·
+     * gpuAllocatorMb) by the harness's sitecustomize, which Python finds on
+     * PYTHONPATH in every process of the container, the kernel included.
+     * CatBoost's GPU pool is a fraction of the card rather than a size, so
+     * the same sitecustomize also needs the card the share was cut from: the
+     * share times the runs on the card, within a few megabytes of what
+     * nvidia-smi said (gpuSliceMb rounds down).
+     */
+    ...(share !== null
+      ? ['-e', `COLLOQ_GPU_MEMORY_MB=${gpuAllocatorMb(share)}`, '-e', `COLLOQ_GPU_CARD_MB=${share * (opts.gpu?.perDevice ?? 1)}`]
+      : []),
+    ...pythonPath(share !== null ? [HARNESS_SITE] : []),
     ...(opts.dependenciesDir ? [...mount(opts.dependenciesDir, '/deps', true), '-e', 'COMP_DEPENDENCIES=/deps'] : []),
     /*
      * The set's environment: `exec`, because its libraries are mapped as code;
@@ -443,10 +528,14 @@ export function runArgs(opts: {
     // in pool.ts).
     `--memory-swap=${memory}`,
     `--cpus=${limits.cpus}`,
+    // The card by its KERNEL_GPUS name; inside the container it is device 0.
+    ...(device ? ['--gpus', `device=${device}`, `--shm-size=${shmMb(limits.memoryMb)}m`] : []),
     `--ulimit=fsize=${fsize}:${fsize}`,
     '--restart=no',
     '--label',
     RUN_LABEL,
+    // The label rooms carry too: by it a new class is kept off the card while this run holds it.
+    ...(device ? ['--label', `colloq.gpu=${device}`] : []),
     '-w',
     '/out',
     '--entrypoint',
@@ -465,6 +554,10 @@ export function runArgs(opts: {
  * open data and the executed notebook do not get here — the teacher's code
  * runs next to the answers, and anything put there may leak into the text of
  * its error.
+ *
+ * Never a card, whatever the notebook had, and there is no way to ask for
+ * one here: a container that holds the hidden answers shares a card with
+ * nobody, and a metric over one CSV has no use for it.
  */
 export function scoreArgs(opts: {
   attemptId?: string
@@ -541,11 +634,185 @@ export function scoreArgs(opts: {
 
 /** What the watchdog over a running container returned. */
 interface Watched {
-  killedBy: 'wall' | 'output' | 'disk' | null
+  killedBy: 'wall' | 'output' | 'disk' | 'gpu' | 'gpu-unread' | null
   progress: RunProgress | null
   exit?: number
 }
 
+/* ------------------------------------------------- a share of a card, outside */
+
+/**
+ * How long one sample of a card may take. nvidia-smi answers in a tenth of a
+ * second; one that has not answered in five is stuck — in the driver, or
+ * starved of the processor by the container it runs in (VramWatch).
+ */
+const VRAM_SAMPLE_MS = 5_000
+
+/**
+ * How long past VRAM_SAMPLE_MS a sample is waited for before it counts as
+ * failed anyway. The shell kills the docker client at VRAM_SAMPLE_MS and
+ * answers when it exits; this is for the day that exit never comes, so that
+ * no client, however stuck, holds the next sample off for the rest of a run.
+ */
+const VRAM_LATE_MS = 1_000
+
+/** How many answered samples a run keeps: the peak is kept apart, and the rule needs two. */
+const VRAM_KEPT = 4
+
+/**
+ * How many samples in a row may fail before a run on a shared card is
+ * stopped as one nobody can watch (VramWatch · unwatchable).
+ *
+ * One failed sample decides nothing: a daemon under load fails an exec now
+ * and then (every other call here is retried for that reason, `retried`),
+ * and two in a row happen. Four in a row is twelve seconds on the status
+ * cycle (STATUS_EVERY_MS), about twenty-five when each hangs to its deadline
+ * (a hung one spans two cycles), in which nothing at all is known of a run
+ * that may hold the whole card: twice what the over-share rule needs to
+ * decide (two samples, three to six seconds) and a hiccup on top. Fewer would
+ * stop honest runs for a slow daemon; more would hand the card, a stretch at
+ * a time, to a run that keeps itself unreadable, and every run sharing the
+ * card would pay for it in out-of-memory errors of their own.
+ */
+export const VRAM_UNREAD_LIMIT = 4
+
+/**
+ * The video memory a run's container holds on its card, MiB, from
+ * `nvidia-smi --query-compute-apps=used_memory --format=csv,noheader,nounits`
+ * run INSIDE it: there it lists that container's processes only (checked on
+ * the lab VM: two containers on one card each see themselves), so the sum is
+ * this run's and nobody else's. No line at all is a valid answer — nothing
+ * on the card yet, 0. `null` — not an answer: a field that is not a whole
+ * number ("[N/A]"), or anything else nvidia-smi might print.
+ */
+export function parseGpuMemory(out: string): number | null {
+  let total = 0
+  for (const raw of out.split('\n')) {
+    const line = raw.trim()
+    if (!line || /^no running processes found$/i.test(line)) continue
+    if (!/^\d+$/.test(line)) return null
+    total += Number(line)
+  }
+  return Number.isSafeInteger(total) ? total : null
+}
+
+/**
+ * The watchdog's rule: whether a run is over its share, given its samples
+ * oldest first (MiB; `null` — a sample that failed or did not parse).
+ *
+ * Over means the last two samples that answered are both above the share
+ * plus its tolerance (competitions/gpu.ts · gpuKillAboveMb). Two, because one
+ * reading may catch a moment — an allocator that grows and gives back within
+ * a second — and a kill is final; two, three seconds apart, are a run that
+ * stays where it should not be. A sample that did not answer decides
+ * nothing either way here: it neither counts toward this kill (a failure is
+ * not a reading) nor resets one (a run that keeps the daemon from answering
+ * must not buy itself the card that way). Failures in a row have a rule of
+ * their own, VRAM_UNREAD_LIMIT.
+ */
+export function gpuOverShare(samples: ReadonlyArray, sliceMb: number): boolean {
+  const limit = gpuKillAboveMb(sliceMb)
+  const answered = samples.filter((sample): sample is number => sample !== null)
+  return answered.length >= 2 && answered[answered.length - 1] > limit && answered[answered.length - 2] > limit
+}
+
+/**
+ * The video memory of one run with a share of a card, watched from outside —
+ * the hard half of the share (the soft one is inside, harness.ts ·
+ * SITECUSTOMIZE, which the participant's code can lift and other libraries
+ * never see).
+ *
+ * The participant is the one watched, so the watch must not be theirs to
+ * stop. nvidia-smi has to run inside the container — only there does it list
+ * that container's processes and nobody else's — and a plain `docker exec`
+ * enters as the container's user: uid 1000, the participant's own, and a
+ * process may signal any other of its uid without a single capability. One
+ * SIGSTOP blinded the first version of this watch for the rest of a run, and
+ * a loop of SIGKILLs left every sample unanswered and the run unstopped
+ * (review, 13 Oct 2026). So the sample runs as root: uid 1000 cannot signal
+ * root without CAP_KILL, and the container has no capability at all
+ * (`--cap-drop=ALL`). Nor does root gain anything in there: no capability, a
+ * read-only root filesystem, and nvidia-smi found on the image's own PATH,
+ * every directory of which is on that read-only root.
+ *
+ * What root cannot buy is the processor: the sample runs in the container's
+ * cgroup, and a notebook that keeps hundreds of threads busy starves it past
+ * its deadline. So no failure is final and none is ignored. Samples are
+ * taken on the watchdog's status cycle, in the background (a slow one must
+ * not hold up the deadline or the progress beacon), never two at once; one
+ * that has not answered by its deadline is given up and the next cycle tries
+ * again; and the samples that fail in a row are counted, VRAM_UNREAD_LIMIT of
+ * them stopping the run (`unwatchable`, watch). A stuck nvidia-smi stays in
+ * the container until it ends, which bounds them at that many as well.
+ */
+class VramWatch {
+  /** The most seen at once, MiB; `null` until a sample answers. */
+  peakMb: number | null = null
+  /** Samples that did not answer since the last one that did. */
+  failures = 0
+  /**
+   * What the last failed sample said, for the teacher: the first line of its
+   * output, or its exit code; `null` — nothing before its deadline.
+   */
+  lastFailure: string | null = null
+  private readonly answered: number[] = []
+  private inFlight = false
+
+  constructor(
+    private readonly container: string,
+    readonly sliceMb: number,
+  ) {}
+
+  sample(): void {
+    if (this.inFlight) return
+    this.inFlight = true
+    const started = Date.now()
+    let late: NodeJS.Timeout | undefined
+    let open = true
+    const settle = (result: { code: number; out: string }): void => {
+      if (!open) return
+      open = false
+      clearTimeout(late)
+      this.inFlight = false
+      this.record(result, Date.now() - started >= VRAM_SAMPLE_MS)
+    }
+    late = setTimeout(() => settle({ code: -1, out: '' }), VRAM_SAMPLE_MS + VRAM_LATE_MS)
+    // As root: see the class's comment. The client itself is killed at VRAM_SAMPLE_MS (shell).
+    docker(
+      ['exec', '--user', '0:0', this.container, 'nvidia-smi', '--query-compute-apps=used_memory', '--format=csv,noheader,nounits'],
+      VRAM_SAMPLE_MS,
+      64 * 1024,
+    ).then(settle, () => settle({ code: -1, out: '' }))
+  }
+
+  private record(result: { code: number; out: string }, stalled: boolean): void {
+    const mb = result.code === 0 ? parseGpuMemory(result.out) : null
+    if (mb === null) {
+      this.failures++
+      this.lastFailure = stalled ? null : result.out.split('\n')[0].trim().slice(0, 200) || `exit ${result.code}`
+      return
+    }
+    this.failures = 0
+    this.peakMb = Math.max(this.peakMb ?? 0, mb)
+    this.answered.push(mb)
+    if (this.answered.length > VRAM_KEPT) this.answered.shift()
+  }
+
+  over(): boolean {
+    return gpuOverShare(this.answered, this.sliceMb)
+  }
+
+  /** Whether the card has gone unread long enough to stop the run (VRAM_UNREAD_LIMIT). */
+  unwatchable(): boolean {
+    return this.failures >= VRAM_UNREAD_LIMIT
+  }
+
+  /** What the teacher reads next to the kill: the last samples that answered. */
+  samples(): number[] {
+    return [...this.answered]
+  }
+}
+
 const INSPECT = '{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}}'
 
 function readJson(file: string): Record | null {
@@ -575,11 +842,15 @@ function num(value: unknown): number | null {
  * Disk growth is bounded by tmpfs itself, including nested directories. The
  * host watchdog handles time/output and reads only bounded progress JSON.
  *
- * `salvage` runs right before a kill for time or printing, while the tmpfs is
- * still there to read: the harness normally stops a late notebook itself and
- * exits in time, and this is for the day it could not — the copy of the
- * notebook it keeps on disk is all that is left to show. A cancellation takes
- * nothing: a withdrawn run publishes nothing.
+ * `salvage` runs right before a kill for time, printing or a card's memory,
+ * while the tmpfs is still there to read: the harness normally stops a late
+ * notebook itself and exits in time, and this is for the day it could not —
+ * the copy of the notebook it keeps on disk is all that is left to show. A
+ * cancellation takes nothing: a withdrawn run publishes nothing.
+ *
+ * `vram` watches a run's share of a card (VramWatch): sampled on the same
+ * status cycle, and a run over its share is killed from outside like one over
+ * its printing — and so is a run whose card cannot be read at all.
  */
 async function watch(
   container: string,
@@ -589,6 +860,7 @@ async function watch(
   onProgress?: (progress: RunProgress) => void,
   signal?: AbortSignal,
   salvage?: () => Promise,
+  vram?: VramWatch | null,
 ): Promise {
   const deadline = Date.now() + limits.wallSeconds * 1000
   let last: RunProgress | null = null
@@ -611,6 +883,8 @@ async function watch(
     // concluded from that — the deadline still bounds the wait.
     if (state === 'running' && (due || Date.now() - lookedInside >= STATUS_EVERY_MS)) {
       lookedInside = Date.now()
+      // Started first and not waited for: it lands while the beacon is read, or by the next look.
+      vram?.sample()
       const snapshot = await docker(['exec', container, 'python', '-I', '/harness/export_files.py', 'status', resultDir], 5000, 128 * 1024)
       let report: any = {}
       try { report = snapshot.code === 0 ? JSON.parse(snapshot.out) : {} } catch { /* incomplete report */ }
@@ -647,6 +921,33 @@ async function watch(
       if (report.complete?.attemptId === attemptId && Number.isInteger(report.complete.exit))
         return { killedBy: null, progress: last, exit: report.complete.exit }
     }
+    /*
+     * Over its share of the card: the other runs on the card are losing their
+     * memory to it right now, and the soft cap inside did not hold it (lifted
+     * by the participant's code, or memory torch never saw). Before the
+     * deadline's own kill: the run has been over its share for a status
+     * cycle at least, and that is what it is stopped for.
+     */
+    if (state === 'running' && vram?.over()) {
+      await salvage?.().catch(() => undefined)
+      await docker(['kill', '--signal=KILL', container], 30_000)
+      return { killedBy: 'gpu', progress: last }
+    }
+    /*
+     * A run on a shared card that nobody can read: fail closed. Whatever
+     * keeps its samples from answering — a notebook starving the container's
+     * processor, a driver that hangs, a daemon that cannot exec — the run may
+     * be holding the whole card meanwhile, and the runs beside it pay. After
+     * the evidence above, so a run seen over its share is told that instead.
+     * No salvage first: what keeps nvidia-smi from answering — most often a
+     * starved processor — would most likely keep the notebook's copy in for
+     * the ten seconds a salvage may take as well, and each of those seconds
+     * may be the card.
+     */
+    if (state === 'running' && vram?.unwatchable()) {
+      await docker(['kill', '--signal=KILL', container], 30_000)
+      return { killedBy: 'gpu-unread', progress: last }
+    }
     if (due) {
       if (state === 'running') await salvage?.().catch(() => undefined)
       await docker(['kill', '--signal=KILL', container], 30_000)
@@ -783,15 +1084,31 @@ async function postmortem(container: string): Promise<{ exit: number | null; oom
  * Printing over the cap is a crash of their notebook (`cell_error`, "NOTEBOOK
  * ERROR"), with a sentence next to it about how much it printed. A write over
  * the cap into the host directory is "answer rejected" (`target_too_large`),
- * because that is exactly what happened.
+ * because that is exactly what happened. A card's memory over the run's share
+ * is out of memory, like the machine's: the same badge, and a sentence of its
+ * own about the share (runner.ts · gpuKillNote).
+ *
+ * A card that could not be read (`gpu-unread`) is named by what was seen
+ * before it went silent. A card that answered and then stopped answering in
+ * this run is a notebook that broke the rule of a shared card — most often by
+ * starving its own container's processor — and its cell fails with our
+ * sentence in place of a traceback (unreadCardNotes); it counts like any
+ * failed notebook, so stopping a run this way cannot be had for free again
+ * and again. A card that never answered at all is our machinery (no
+ * nvidia-smi in the container, a driver in trouble), never the participant's
+ * doing: a harness error, which blames nobody and costs them nothing.
  */
 export function verdictOfRun(input: {
   oom: boolean
-  killedBy: 'wall' | 'output' | 'disk' | null
+  killedBy: 'wall' | 'output' | 'disk' | 'gpu' | 'gpu-unread' | null
   status: string | null
   produced: boolean
+  /** Whether the run's card answered at least once (VramWatch); only `gpu-unread` asks. */
+  cardRead?: boolean
 }): RunVerdict {
   if (input.oom) return 'out-of-memory'
+  if (input.killedBy === 'gpu') return 'out-of-memory'
+  if (input.killedBy === 'gpu-unread') return input.cardRead ? 'cell_error' : 'harness_error'
   if (input.killedBy === 'wall') return 'timeout'
   if (input.killedBy === 'output') return 'cell_error'
   if (input.killedBy === 'disk') return 'target_too_large'
@@ -832,6 +1149,41 @@ function asVerdict(status: string): RunVerdict {
   return VERDICTS.has(status) ? (status as RunVerdict) : 'unknown'
 }
 
+/**
+ * The words for a run stopped because its card could not be read (watch ·
+ * `gpu-unread`): `participant` goes where a traceback would (RunOutcome.detail,
+ * after runner.ts's own "failed at cell" head), `teacher` heads the log.
+ *
+ * Both are ours alone. Every number in them was counted outside the notebook
+ * (the samples, the share), and there is no cell number in the participant's
+ * sentence: the head that precedes it names the cell, clamped to the notebook
+ * that was sent (runner.ts · clampCells), and a cell read from the beacon
+ * here would not be. The reason in the teacher's line is what docker or
+ * nvidia-smi said, running as root where the notebook cannot reach. A card
+ * that never answered has no participant sentence at all — that run ends as
+ * our own failure (verdictOfRun) — and the teacher is told where to look.
+ */
+export function unreadCardNotes(input: {
+  device: string
+  sliceMb: number
+  failures: number
+  /** What the last failed sample said; `null` — nothing before its deadline. */
+  lastFailure: string | null
+  /** Whether the card answered at least once in this run. */
+  answered: boolean
+}): { participant: string | null; teacher: string } {
+  const reason = input.lastFailure ?? tr('competitions.answer.gpuSampleStalled', { seconds: VRAM_SAMPLE_MS / 1000 })
+  const teacher = tr('competitions.answer.gpuUnreadKilled', {
+    device: input.device,
+    count: input.failures,
+    reason,
+    slice: input.sliceMb,
+  })
+  return input.answered
+    ? { participant: tr('competitions.answer.gpuUnread', { count: input.failures }), teacher }
+    : { participant: null, teacher: `${teacher} ${tr('competitions.answer.gpuNeverRead')}` }
+}
+
 /**
  * Why the package set failed to install, when the harness could tell — read
  * off its run.json by both runners, so that both give the participant the
@@ -864,11 +1216,19 @@ class DockerCompetitionRunner implements CompetitionRunner {
       packagesMb: request.packagesMb,
       limits: request.limits,
       target: SUBMISSION_NAME,
+      gpu: request.gpu,
     })
     const started_ = await docker(args, 120_000)
-    const held = request.limits.memoryMb + CONTAINER_CUSHION_MB
+    const held = { memoryMb: request.limits.memoryMb + CONTAINER_CUSHION_MB, gpu: request.gpu?.device || null }
     if (started_.code !== 0) {
       await removeContainer(request.container, held)
+      /*
+       * The most common trouble of a card is not ours: the host lacks
+       * nvidia-container-toolkit, and docker answers "could not select device
+       * driver". The teacher reads the same hint a GPU room gives, and only
+       * when docker said exactly that.
+       */
+      const noDriver = held.gpu !== null && /device driver|nvidia/i.test(started_.out)
       return {
         status: 'harness_error',
         cell: -1,
@@ -876,15 +1236,17 @@ class DockerCompetitionRunner implements CompetitionRunner {
         wall: Date.now() - started,
         submission: null,
         detail: '',
-        log: started_.out,
+        log: noDriver ? `${started_.out}${tr('server.theHostMayBeMissingNvidiaContainer.f5c0df')}` : started_.out,
         diagnostics: { exit: null, oomKilled: false, backend: this.backend },
       }
     }
+    const share = heldShareMb(request.gpu)
+    const vram = share === null ? null : new VramWatch(request.container, share)
     let watched: Watched = { killedBy: null, progress: null }
     let dead = { exit: null as number | null, oom: false, tail: '' }
     try {
       const salvage = () => collectExports(request.container, '/result', request.resultDir, [[EXECUTED_FILE, EXECUTED_BYTES]], SALVAGE_MS)
-      watched = await watch(request.container, '/result', request.limits, attemptId, request.onProgress, request.signal, salvage)
+      watched = await watch(request.container, '/result', request.limits, attemptId, request.onProgress, request.signal, salvage, vram)
       dead = await postmortem(request.container)
       if (watched.exit !== undefined) {
         dead.exit = watched.exit
@@ -902,7 +1264,17 @@ class DockerCompetitionRunner implements CompetitionRunner {
     const answer = path.join(request.resultDir, SUBMISSION_NAME)
     const produced = competitionsFs.existsSync(answer)
     const status = dead.exit !== 0 && report?.status === 'ok' ? 'exit' : typeof report?.status === 'string' ? report.status : null
-    const verdict = verdictOfRun({ oom: dead.oom, killedBy: watched.killedBy, status, produced })
+    const cardRead = vram !== null && vram.peakMb !== null
+    const verdict = verdictOfRun({ oom: dead.oom, killedBy: watched.killedBy, status, produced, cardRead })
+    const unread = watched.killedBy === 'gpu-unread' && vram
+      ? unreadCardNotes({
+        device: request.gpu?.device ?? '',
+        sliceMb: vram.sliceMb,
+        failures: vram.failures,
+        lastFailure: vram.lastFailure,
+        answered: cardRead,
+      })
+      : null
     const beat = watched.progress
     return {
       status: verdict,
@@ -910,8 +1282,9 @@ class DockerCompetitionRunner implements CompetitionRunner {
       cells: num(report?.cells) ?? beat?.cells ?? 0,
       wall: Date.now() - started,
       submission: verdict === 'ok' && produced ? answer : null,
-      detail: typeof report?.detail === 'string' ? report.detail : '',
-      log: dead.tail,
+      // The sentence goes only where it is read: after a cell's head (runner.ts · notebookNote).
+      detail: verdict === 'cell_error' && unread?.participant ? unread.participant : typeof report?.detail === 'string' ? report.detail : '',
+      log: unread ? [unread.teacher, dead.tail].filter(Boolean).join('\n') : dead.tail,
       ...dependencyFailureOf(verdict, report, request.packagesMb ?? request.limits.tmpfsMb),
       diagnostics: {
         exit: dead.exit,
@@ -919,6 +1292,8 @@ class DockerCompetitionRunner implements CompetitionRunner {
         backend: this.backend,
         killedBy: watched.killedBy ?? undefined,
         peakBytes: num(report?.peakBytes),
+        // Measured from outside, for the teacher; only a watched run has them.
+        ...(vram ? { gpuPeakMb: vram.peakMb, gpuSamplesMb: vram.samples() } : {}),
       },
     }
   }
@@ -942,7 +1317,8 @@ class DockerCompetitionRunner implements CompetitionRunner {
       splitSeed: request.competition.splitSeed,
     })
     const launched = await docker(args, 120_000)
-    const held = request.limits.memoryMb + CONTAINER_CUSHION_MB
+    // Memory only: the metric's container never has a card.
+    const held = { memoryMb: request.limits.memoryMb + CONTAINER_CUSHION_MB }
     if (launched.code !== 0) {
       await removeContainer(request.container, held)
       return metricFailure(this.backend, launched.out, Date.now() - started, null, false)
@@ -1008,14 +1384,23 @@ class DockerCompetitionRunner implements CompetitionRunner {
    * `competition-run`/`competition-score` are created only by this module and
    * live only for the duration of one submission, so any found after startup
    * is already orphaned.
+   *
+   * Each is read with the card it holds (its `colloq.gpu` label): an orphan
+   * whose removal fails keeps that card out of the queue's reach until the
+   * retry gets it, exactly like a container of this life (capacity ·
+   * heldGpus). The queue rows of the previous life gave their cards back
+   * when they were put back into waiting (store.ts · reclaimQueue).
    */
   async sweep(legacyContainers: readonly string[] = []): Promise {
     let dropped = 0
     for (const label of [RUN_LABEL, SCORE_LABEL]) {
-      const found = await docker(['ps', '-aq', '--filter', `label=${label}`, '--filter', `label=${INSTANCE_LABEL}`], 20_000)
+      const found = await docker(['ps', '-a', '--filter', `label=${label}`, '--filter', `label=${INSTANCE_LABEL}`, '--format', '{{.ID}}\t{{.Label "colloq.gpu"}}'], 20_000)
       if (found.code !== 0) continue
-      for (const id of found.out.split('\n').map((line) => line.trim()).filter(Boolean)) {
-        if (await removeContainer(id)) dropped++
+      for (const line of found.out.split('\n').map((entry) => entry.trim()).filter(Boolean)) {
+        const [id, card = ''] = line.split('\t').map((part) => part.trim())
+        // No label prints nothing or ""; a device id has no blanks in it.
+        const gpu = card && card !== '' && !/\s/.test(card) ? card : null
+        if (await removeContainer(id, { gpu })) dropped++
       }
     }
     // Old queue rows name specific pre-scope containers. Never scan/delete all
@@ -1054,7 +1439,8 @@ class DockerCompetitionRunner implements CompetitionRunner {
    * Mac, and on colima that is twelve gigabytes out of the machine's
    * thirty-six. Counting by the host would mean taking on a submission that
    * `docker run` will refuse in the middle of a class. Containers still
-   * awaiting removal keep their own memory out of reach, and nothing more.
+   * awaiting removal keep their own memory out of reach, and nothing more —
+   * and their cards, a share each (`heldGpus`).
    */
   async capacity(): Promise {
     // Not awaited: the pump asks this with admission held, and a daemon that
@@ -1062,13 +1448,18 @@ class DockerCompetitionRunner implements CompetitionRunner {
     // Until they go, the stuck ones hold their memory, as counted below.
     if (pendingCleanup.size > 0) void retryPendingCleanup()
     let heldMb = 0
-    for (const mb of pendingCleanup.values()) heldMb += mb
+    const heldGpus: Record = {}
+    for (const held of pendingCleanup.values()) {
+      heldMb += held.memoryMb
+      if (held.gpu) heldGpus[held.gpu] = (heldGpus[held.gpu] ?? 0) + 1
+    }
     const memory = await competitionMemory()
     return {
       availableMb: memory.availableMb === null ? null : Math.max(0, memory.availableMb - heldMb),
       usableMb: memory.usableMb,
       memAvailableMb: memory.memAvailableMb,
       heldMb,
+      heldGpus,
     }
   }
 }
diff --git a/server/src/competitions/fake-runner.ts b/server/src/competitions/fake-runner.ts
index 14479acc..0e61f673 100644
--- a/server/src/competitions/fake-runner.ts
+++ b/server/src/competitions/fake-runner.ts
@@ -37,7 +37,10 @@
  * `"detailFrom": "test.csv"` raises with them (`ValueError: `), the
  * way a notebook printing the test into its traceback would; `"executed": true`
  * leaves an executed notebook whose output is what was read. On the metric
- * side `"echo": true` has a refusing metric quote the answer back.
+ * side `"echo": true` has a refusing metric quote the answer back. And
+ * `"killedBy": "gpu"` next to `"status": "oom"` is a run the watchdog stopped
+ * for going past its share of a card (docker-runner.ts · VramWatch): what
+ * its neighbours on the card are judged against (runner.ts · neighbourOverran).
  *
  * The stability of the number is not decoration. The leaderboard, "the
  * participant's best submission" and the final rescoring after the deadline
@@ -181,6 +184,20 @@ export class FakeCompetitionRunner implements CompetitionRunner {
   /** The machine's MemAvailable for the admission floor; `null` — unknown, not checked. */
   memAvailableMb: number | null = null
 
+  /**
+   * Cards held by containers "still awaiting removal", card → how many
+   * (Capacity.heldGpus): the queue must count them as runs on the card
+   * without a daemon that fails a `docker rm`.
+   */
+  heldGpus: Record = {}
+
+  /**
+   * The notebook runs asked of it, newest last, with the card each was given
+   * (`null` — none): which card the queue chose is visible only here, at the
+   * moment of the run. Bounded: the test stand lives for days.
+   */
+  readonly notebookRuns: Array<{ submissionId: string; container: string; gpu: RunRequest['gpu'] | null }> = []
+
   /** Runs in progress: container name → how to cut it short. */
   private readonly live = new Map void>()
 
@@ -189,6 +206,8 @@ export class FakeCompetitionRunner implements CompetitionRunner {
 
   async run(request: RunRequest): Promise {
     const started = Date.now()
+    this.notebookRuns.push({ submissionId: request.submissionId, container: request.container, gpu: request.gpu ?? null })
+    if (this.notebookRuns.length > 256) this.notebookRuns.shift()
     const notebook = readFile(path.join(request.inputDir, NOTEBOOK_FILE))
     const asked = notebook ? directiveOf(notebook) : {}
     const cells = notebook ? cellsOf(notebook) : 0
@@ -224,8 +243,10 @@ export class FakeCompetitionRunner implements CompetitionRunner {
         started,
         detail: raised ? `Traceback (most recent call last):\n  Cell In[1], line 1\nValueError: ${raised.toString('utf8')}` : typeof asked.detail === 'string' ? asked.detail : '',
         log: typeof asked.log === 'string' ? asked.log : '',
-        oom: status === 'out-of-memory',
+        // A run the watchdog stopped for its share of a card (`"killedBy": "gpu"`) is no OOM of the machine.
+        oom: status === 'out-of-memory' && asked.killedBy !== 'gpu',
         exit: status === 'timeout' ? 137 : 1,
+        killedBy: asked.killedBy === 'gpu' ? 'gpu' : undefined,
       })
     }
 
@@ -374,7 +395,7 @@ export class FakeCompetitionRunner implements CompetitionRunner {
   }
 
   async capacity(): Promise {
-    return { availableMb: this.availableMb, memAvailableMb: this.memAvailableMb }
+    return { availableMb: this.availableMb, memAvailableMb: this.memAvailableMb, heldGpus: { ...this.heldGpus } }
   }
 
   /**
@@ -414,7 +435,7 @@ export class FakeCompetitionRunner implements CompetitionRunner {
       log?: string
       oom?: boolean
       exit?: number
-      killedBy?: 'wall' | 'output' | 'disk'
+      killedBy?: 'wall' | 'output' | 'disk' | 'gpu'
     },
   ): RunOutcome {
     return {
diff --git a/server/src/competitions/forecast.ts b/server/src/competitions/forecast.ts
index b6ba4df5..772fa30b 100644
--- a/server/src/competitions/forecast.ts
+++ b/server/src/competitions/forecast.ts
@@ -11,10 +11,23 @@
  * work (panel.ts · fairOrder): "third" must mean the same on both screens.
  * Jobs postponed by a resource retry keep their row but get neither a place
  * nor an estimate: they cannot take a slot yet.
+ *
+ * GPU WORK. A notebook with a card needs a general slot AND a share of a
+ * runner card, and while no share is free the queue passes it over and takes
+ * the work behind it (store.ts · selectNext). The simulation does the same
+ * (panel.ts · queueEtas), so the place is the order of the simulated starts:
+ * a GPU notebook first in the order but waiting for a card is not "first",
+ * and the CPU work that will start before it is not behind it. With no GPU
+ * work waiting that is exactly the order. Under the cluster the queue keeps
+ * no card, but it keeps its GPU notebooks to the number the broker has units
+ * for (gpu.ts · clusterGpuSlots), and the simulation counts that number as
+ * the cards' places.
  */
-import { expectedRunMs, fairOrder, queueEtas } from './panel.js'
+import { expectedRunMs, fairOrder, queueEtas, type Eta } from './panel.js'
 import { METRIC_WALL_SECONDS } from './runner-port.js'
 import { getCompetition, queueRows, recentDurations, recentMetricRuns, type QueueRow } from './store.js'
+import { clusterGpuSlots, runnerCardsHere, runnerGpuDevices, runnerGpuPerDevice, runnerGpuState, sliceOverCards } from './gpu.js'
+import { gpuCards } from '../kernel/resources.js'
 
 /** A competition's "recent" runs: a competition lives for weeks, and its first runs solved a different task. */
 const RECENT_RUNS = 20
@@ -23,7 +36,10 @@ const INSTANCE_RUNS = 50
 
 export interface Forecast {
   running: QueueRow[]
-  /** Claimable waiting jobs, in the order they will be taken. */
+  /**
+   * Claimable waiting jobs, in the order they will be taken: by place, and
+   * after them the GPU work that has no card to start on at all.
+   */
   eligible: QueueRow[]
   /** Postponed by a resource retry: visible, but not claimable yet. */
   deferred: QueueRow[]
@@ -31,23 +47,40 @@ export interface Forecast {
   at: Map
 }
 
-export function queueForecast(slots: number, now = Date.now()): Forecast {
+/**
+ * `gpuSlots` — how many runs may hold a card share at once (under the
+ * cluster, a place among its units); `null` — no card is counted at all, and
+ * GPU work is simulated like any other. By default what the screens show
+ * (forecastGpuSlots).
+ */
+export function queueForecast(slots: number, now = Date.now(), gpuSlots: number | null = forecastGpuSlots(now)): Forecast {
   const rows = queueRows()
   const running = rows.filter((row) => row.state === 'running')
   const waiting = rows.filter((row) => row.state === 'waiting')
   const busy = new Set(running.map((row) => row.entrantId))
-  const eligible = fairOrder(waiting.filter((row) => row.notBefore <= now), busy)
+  const preferred = fairOrder(waiting.filter((row) => row.notBefore <= now), busy)
   const deferred = fairOrder(waiting.filter((row) => row.notBefore > now), busy)
   const expected = runEstimator()
+  const needsCard = cardNeeds()
+  // Under the cluster a running GPU notebook holds no card of ours, but it does hold one of the cluster's places.
+  const cluster = !runnerCardsHere()
   const etas = queueEtas({
     slots,
     running: running.map((row) => ({
       expectedMs: expected(row),
       elapsedMs: row.startedAt === null ? 0 : Math.max(0, now - row.startedAt),
+      gpu: row.gpu !== null || (cluster && needsCard(row)),
     })),
-    waiting: eligible.map((row) => ({ expectedMs: expected(row) })),
+    waiting: preferred.map((row) => ({ expectedMs: expected(row), gpu: needsCard(row) })),
+    ...(gpuSlots === null ? {} : { gpuSlots }),
   })
-  const at = new Map(eligible.map((row, index) => [row.submissionId, { place: index + 1, ...etas[index] }] as const))
+  // Places by start; among jobs starting at one moment, the queue's order.
+  const starting = preferred
+    .map((row, index) => ({ row, index, eta: etas[index] }))
+    .filter((entry): entry is { row: QueueRow; index: number; eta: Eta } => entry.eta !== null)
+    .sort((a, b) => a.eta.startMs - b.eta.startMs || a.index - b.index)
+  const at = new Map(starting.map((entry, place) => [entry.row.submissionId, { place: place + 1, ...entry.eta }] as const))
+  const eligible = [...starting.map((entry) => entry.row), ...preferred.filter((row) => !at.has(row.submissionId))]
   return { running, eligible, deferred, at }
 }
 
@@ -82,3 +115,124 @@ function runEstimator(): (row: QueueRow) => number {
     return known
   }
 }
+
+/**
+ * Whether a waiting row needs a card share: a NOTEBOOK run of a competition
+ * whose card is on — the rule of store.ts · NEEDS_CARD, read here off the
+ * competition, once per competition per forecast.
+ */
+function cardNeeds(): (row: QueueRow) => boolean {
+  const byCompetition = new Map()
+  return (row) => {
+    if (row.kind !== 'notebook') return false
+    let needs = byCompetition.get(row.competitionId)
+    if (needs === undefined) {
+      needs = getCompetition(row.competitionId)?.limits.gpu === true
+      byCompetition.set(row.competitionId, needs)
+    }
+    return needs
+  }
+}
+
+/* ------------------------------------------------- the cards on the screens */
+
+/** The runner cards as both screens show them. */
+export interface RunnerGpuPicture {
+  /**
+   * How many GPU runs may go at once: the cards given to submissions times
+   * the runs per card, less the cards a class still holds (and every card
+   * while docker cannot say who holds them) — what the queue takes by.
+   */
+  slots: number
+  /** The share every run is promised, MB: the smallest card's; `null` — unknown. */
+  sliceMb: number | null
+  /**
+   * Each card with the runs it may hold now (gpu.ts · runnerGpuState): what
+   * the places taken are counted against (gpu.ts · placesTaken). A run shown
+   * on a card carries its own share (QueueRow.gpuSliceMb), not one read here.
+   */
+  cards: ReadonlyArray<{ id: string; capacity: number }>
+}
+
+/** How long a count of the cards is drawn from before it is made again. */
+const PICTURE_TTL_MS = 2_000
+
+let picture: { key: string; at: number; value: RunnerGpuPicture } | null = null
+let picturing: Promise | null = null
+
+/** The cards as configured right now: a count made for other cards is not this one. */
+function pictureKey(devices: readonly string[], perDevice: number): string {
+  return `${devices.join(',')}|${perDevice}`
+}
+
+/**
+ * The runner cards for the screens; `null` — no card is given to
+ * submissions here, or the cards are the cluster's.
+ *
+ * Drawn from the last count, and counted again behind the answer once that
+ * is two seconds old: the console streams its state every second and a half,
+ * every participant's page polls, and an honest count of the slots is a
+ * census of the rooms' containers — a docker call per look would be one per
+ * student per poll. Until the first count, and for a moment after the owner
+ * changes the cards, the picture is the plain product of cards and runs per
+ * card, with the share unknown: a number for a second, never a made-up size.
+ */
+export function runnerGpuPicture(now = Date.now()): RunnerGpuPicture | null {
+  if (!runnerCardsHere()) return null
+  const devices = runnerGpuDevices()
+  if (devices.length === 0) return null
+  const perDevice = runnerGpuPerDevice()
+  const key = pictureKey(devices, perDevice)
+  if (picture === null || picture.key !== key || now - picture.at >= PICTURE_TTL_MS) void startCount()
+  if (picture?.key === key) return picture.value
+  return { slots: devices.length * perDevice, sliceMb: null, cards: devices.map((id) => ({ id, capacity: perDevice })) }
+}
+
+/**
+ * How many runs may hold a card share at once, as the forecast counts them:
+ * none without runner cards (a GPU notebook there starts nowhere). Under the
+ * cluster, the GPU notebooks the queue lets ask it at once (gpu.ts ·
+ * clusterGpuSlots): simulated against every slot, as they were, a class of
+ * GPU notebooks read "≈ 2 min" for a wait of an hour, and the CPU work behind
+ * them read the same lie the other way.
+ */
+export function forecastGpuSlots(now = Date.now()): number | null {
+  if (!runnerCardsHere()) return clusterGpuSlots()
+  return runnerGpuPicture(now)?.slots ?? 0
+}
+
+/**
+ * Count the cards now and keep the count; resolves when it is kept. A count
+ * already under way may be for the cards as they were, so it is waited out
+ * and a new one made. Never rejects.
+ */
+export async function refreshRunnerGpuPicture(): Promise {
+  while (picturing) await picturing
+  await startCount()
+}
+
+/** One count at a time, whoever asks. */
+function startCount(): Promise {
+  picturing ??= countCards()
+    .catch(() => undefined)
+    .finally(() => {
+      picturing = null
+    })
+  return picturing
+}
+
+async function countCards(): Promise {
+  const devices = runnerGpuDevices()
+  if (devices.length === 0) return
+  const perDevice = runnerGpuPerDevice()
+  const [state, cards] = await Promise.all([runnerGpuState(), gpuCards(devices)])
+  picture = {
+    key: pictureKey(devices, perDevice),
+    at: Date.now(),
+    value: {
+      slots: state.slots,
+      sliceMb: sliceOverCards(cards, perDevice),
+      cards: state.cards.map((card) => ({ id: card.id, capacity: card.capacity })),
+    },
+  }
+}
diff --git a/server/src/competitions/gpu.ts b/server/src/competitions/gpu.ts
new file mode 100644
index 00000000..3177a1d0
--- /dev/null
+++ b/server/src/competitions/gpu.ts
@@ -0,0 +1,459 @@
+/**
+ * Video cards for competition submissions: which cards are given to them, how
+ * many runs share one, how much video memory each run is promised, and which
+ * card the next GPU run takes.
+ *
+ * WHY WHOLE CARDS, TAKEN OUT OF THE ROOMS' POOL. Until 13 Oct 2026 a
+ * competition on `base-gpu` ran its CUDA torch on the CPU without a word to
+ * anyone: the submission container had no device at all. Giving it one is the
+ * easy half; the hard half is that docker does not refuse a second container
+ * on a card, and a card shared by a lecture and a submission has no wall
+ * between their video memory: a participant's runaway training kills the
+ * lecture's. So the owner gives whole cards (entries of KERNEL_GPUS) to
+ * submissions, and such a card leaves the rooms' pool (kernel/pool.ts ·
+ * roomGpuDevices). Rooms and submissions never meet on a card.
+ *
+ * WHY A SHARE, AND WHY IT IS ENFORCED. Several submissions may time-share one
+ * card (`perDevice`, 1 by default). Each is then promised the card's memory
+ * divided among them, and the panel may promise it only because it is held
+ * (the honesty rule of web/src/admin/ui/Resources.svelte): torch's allocator is
+ * capped a little below the share (GPU_CONTEXT_MB: the CUDA context lives
+ * outside it), and the docker runner kills a container that holds more than
+ * the share plus a tolerance on two samples in a row (GPU_TOLERANCE_MB). With
+ * one run per card the run has the whole card, and nothing is capped. A card
+ * that cannot say its memory gets no share at all — no cap, no watchdog — and
+ * the panel says the size is unknown instead of making one up.
+ *
+ * WHAT IS NOT HERE: the setting itself (competitions/settings.ts, which the
+ * pool can import without a cycle), and the queue: which job waits for a card
+ * is decided where the job is taken (store.ts · takeNext), with
+ * `runnerGpuState`, `cardsInUse` and `pickRunnerGpu` below as its numbers and
+ * its rule. Under Kubernetes none of this applies: the cluster defines the
+ * card share (RUNTIME_COMPETITION_GPU_RESOURCE), and the panel neither lists
+ * nor gives cards there; the queue only keeps the number of GPU notebooks the
+ * broker says it has units for (`clusterGpuSlots`).
+ */
+import { gpuAssignments, gpuDevices, submissionGpuCards, submissionsUseLocalCards } from '../kernel/pool.js'
+import { gpuCards } from '../kernel/resources.js'
+import { kernelBackend } from '../kernel/runtime-client.js'
+import { needsGpu } from '../environments.js'
+import { gpuPerDeviceSetting, resolveGpuPerDevice, resolveRunnerGpus, runnerGpuSetting } from './settings.js'
+import { SETTINGS_LIMITS, type CompetitionGpuSettings } from '@shared/competitions-settings'
+import { COMPETITION_GPU_SLOTS_MAX } from '@shared/competition-runtime'
+
+/* ---------------------------------------------------------------- numbers */
+
+/**
+ * What the CUDA context takes outside torch's allocator, MB: measured at
+ * ~458 MB on an RTX 4090 with torch 2.14 / CUDA 13.0, rounded up. The
+ * allocator's cap is the share minus this, so that the context and the
+ * allocations together stay inside the share.
+ */
+export const GPU_CONTEXT_MB = 512
+
+/**
+ * How far past its share a run may be seen before it is killed, MB: the
+ * context is not the same size on every card and driver, and a watchdog that
+ * kills an honest run for 30 MB of driver bookkeeping is worse than none.
+ */
+export const GPU_TOLERANCE_MB = 256
+
+/** The allocator never gets less than this, however small the share: a run must be able to start. */
+export const GPU_ALLOCATOR_FLOOR_MB = 256
+
+/**
+ * How long an answer waits for a card nobody has asked about yet; after that
+ * the size is unknown on this answer, and known on the next. A card asked
+ * about before is answered from memory at once (kernel/resources.ts ·
+ * gpuCards), so this is the wait of a card never asked about, and nothing else.
+ *
+ * The same short wait for the owner's settings form. It used to wait eight
+ * seconds, being where the sizes are read, and on the first open after a
+ * restart that was the whole Resources tab blank for the probe. The server
+ * now asks every card from its start (kernel/resources.ts · watchGpuCards),
+ * so a card is never asked about only in the seconds before that first
+ * answer — a form opened then joins the probe for this long, and asks again
+ * by itself while a size is unknown (web/src/admin/runner-cards.ts ·
+ * sizesUnknown).
+ */
+const CARD_WAIT_MS = 1_500
+
+/**
+ * A run's share of a card, MB: the card's memory over the runs sharing it,
+ * rounded down. `null` when the card's memory is unknown: nothing is promised
+ * then, and nothing enforced.
+ */
+export function gpuSliceMb(cardMb: number | null, perDevice: number): number | null {
+  if (cardMb === null || !Number.isFinite(cardMb) || cardMb <= 0) return null
+  return Math.floor(cardMb / Math.max(1, Math.floor(perDevice)))
+}
+
+/** The cap given to torch's allocator for a share (COLLOQ_GPU_MEMORY_MB): the share less the context, never below the floor. */
+export function gpuAllocatorMb(sliceMb: number): number {
+  return Math.max(GPU_ALLOCATOR_FLOOR_MB, sliceMb - GPU_CONTEXT_MB)
+}
+
+/** Above this many MB, seen twice in a row, the docker runner kills the run from outside. */
+export function gpuKillAboveMb(sliceMb: number): number {
+  return sliceMb + GPU_TOLERANCE_MB
+}
+
+/**
+ * The smallest share over these cards — what every run is promised whichever
+ * card it lands on. `null` when there is no card, or when any of them did not
+ * say its size: a promise that holds on some cards only is not one.
+ */
+export function sliceOverCards(cards: ReadonlyArray<{ memoryMb: number | null }>, perDevice: number): number | null {
+  let least: number | null = null
+  for (const card of cards) {
+    const slice = gpuSliceMb(card.memoryMb, perDevice)
+    if (slice === null) return null
+    least = least === null ? slice : Math.min(least, slice)
+  }
+  return least
+}
+
+/* ------------------------------------------------------------ the cards */
+
+/**
+ * Whether submissions run on the cards of this machine at all: the docker
+ * runner and its test stand-in do; under the broker the cluster has them.
+ * The pool's own rule (kernel/pool.ts · submissionsUseLocalCards), so that the
+ * cards it withholds from rooms and the cards given here never disagree.
+ */
+export function runnerCardsHere(): boolean {
+  return submissionsUseLocalCards()
+}
+
+/**
+ * How many GPU notebooks of this instance may run at once under the cluster:
+ * what the broker reports as competition.gpuSlots (its units of the GPU
+ * resource for submissions), noted by the pump on every pass (runner.ts ·
+ * pumpOnce) and read by the forecast between passes.
+ *
+ * One until the broker says, and whenever it says nothing usable (a broker
+ * from before the number, or no GPU resource at all): the queue then lets one
+ * GPU notebook ask the cluster at a time, which is the most a cluster that
+ * did not say can be trusted with. Without a number the queue sent every GPU
+ * notebook in its turn, each held a general slot while its Pod waited for a
+ * unit, and the CPU work behind them waited for hours.
+ */
+let clusterSlots = 1
+
+export function noteClusterGpuSlots(reported: unknown): number {
+  clusterSlots = typeof reported === 'number' && Number.isSafeInteger(reported) && reported > 0
+    ? Math.min(COMPETITION_GPU_SLOTS_MAX, reported)
+    : 1
+  return clusterSlots
+}
+
+export function clusterGpuSlots(): number {
+  return clusterSlots
+}
+
+/**
+ * The cards given to submissions in effect: the chosen ones that KERNEL_GPUS
+ * still lists, in KERNEL_GPUS order. A card the operator removed from
+ * KERNEL_GPUS may not be on the machine any more; it is dropped here, not
+ * handed to `docker run`.
+ */
+export function runnerGpuDevices(): string[] {
+  if (!runnerCardsHere()) return []
+  const chosen = new Set(runnerGpuSetting().devices)
+  // Each card once, even if KERNEL_GPUS names it twice: twice would double its runs.
+  return [...new Set(gpuDevices())].filter((device) => chosen.has(device))
+}
+
+/** How many runs share one runner card at a time. */
+export function runnerGpuPerDevice(): number {
+  return gpuPerDeviceSetting().value
+}
+
+/**
+ * The share every GPU run is promised now, MB (`sliceOverCards` over the
+ * runner cards); `null` — no card, a card of unknown size, or the cluster's
+ * business. `waitMs` bounds the wait for a card asked about for the first time.
+ */
+export async function runnerSliceMb(options: { waitMs?: number } = {}): Promise {
+  const devices = runnerGpuDevices()
+  if (devices.length === 0) return null
+  return sliceOverCards(await gpuCards(devices, options), runnerGpuPerDevice())
+}
+
+/**
+ * The share of a run on THIS card, MB — what the run itself is capped to
+ * (RunRequest.gpu.sliceMb). Per card, not the smallest: the panel promises "at
+ * least", and a run on the bigger card gets its bigger share. `perDevice` —
+ * the runs per card the run was taken at (QueueRow.gpuPerDevice); without it,
+ * the number in force.
+ */
+export async function gpuSliceFor(device: string, options: { waitMs?: number; perDevice?: number } = {}): Promise {
+  const [card] = await gpuCards([device], { waitMs: options.waitMs })
+  return gpuSliceMb(card?.memoryMb ?? null, options.perDevice ?? runnerGpuPerDevice())
+}
+
+/**
+ * What a competition's page says next to its limits: the share a run gets,
+ * or `null` for a competition on the CPU and for a size nobody knows.
+ */
+export async function competitionGpuSliceMb(competition: { limits: { gpu?: boolean } }): Promise {
+  if (competition.limits.gpu !== true) return null
+  return runnerSliceMb({ waitMs: CARD_WAIT_MS })
+}
+
+/**
+ * Whether an environment needs a GPU, and "no" instead of an exception: under
+ * the broker an environment missing from the catalog throws, and a form's
+ * refusal must name the problem, not fail on it.
+ */
+export function environmentNeedsGpu(name: string): boolean {
+  try {
+    return needsGpu(name)
+  } catch {
+    return false
+  }
+}
+
+/* -------------------------------------------------------- the queue's numbers */
+
+/** One runner card as the queue sees it. */
+export interface RunnerGpuCard {
+  id: string
+  /**
+   * How many runs it may hold at once: `perDevice`, or 0 while a room's
+   * container still holds it (a card given to submissions stays with the class
+   * that had it until its container ends — kernel/pool.ts · roomGpuDevices).
+   */
+  capacity: number
+  /** Runs holding it now, by the queue's own count. */
+  busy: number
+}
+
+export interface RunnerGpuState {
+  cards: RunnerGpuCard[]
+  perDevice: number
+  /** How many GPU runs may go at once: the sum of the capacities. */
+  slots: number
+  /** How many hold a card now. */
+  busy: number
+  /**
+   * Whether docker answered which cards the rooms' containers hold. When it
+   * did not, every card counts as held (capacity 0): a GPU run waits for an
+   * answer rather than landing on a lecture's card.
+   */
+  census: boolean
+}
+
+/** Pure: the cards with their capacities, from the rooms' hold on them (`null` — unknown, so all held). */
+export function runnerGpuCards(input: {
+  devices: readonly string[]
+  perDevice: number
+  heldByRooms: ReadonlySet | null
+  busy?: ReadonlyMap
+}): RunnerGpuCard[] {
+  return input.devices.map((id) => ({
+    id,
+    capacity: input.heldByRooms === null || input.heldByRooms.has(id) ? 0 : Math.max(1, Math.floor(input.perDevice)),
+    busy: input.busy?.get(id) ?? 0,
+  }))
+}
+
+/**
+ * The cards rooms' containers hold right now, by their `colloq.gpu` labels,
+ * stopped containers included (they come back with the same card); `null` —
+ * docker did not answer. Without docker rooms there are no room containers at
+ * all, so nothing is held.
+ */
+export async function roomHeldCards(): Promise | null> {
+  if (kernelBackend() !== 'docker') return new Set()
+  try {
+    return new Set((await gpuAssignments({ strict: true })).map((held) => held.gpu).filter((gpu) => gpu.length > 0))
+  } catch {
+    return null
+  }
+}
+
+/**
+ * The runner cards now: each card's capacity (`perDevice`, or 0 while a room
+ * holds it) and how many runs hold it. `busy` is the queue's count per card
+ * (its running rows); `held` replaces the census, for tests.
+ */
+export async function runnerGpuState(options: {
+  busy?: ReadonlyMap
+  held?: () => Promise | null>
+} = {}): Promise {
+  const devices = runnerGpuDevices()
+  const perDevice = runnerGpuPerDevice()
+  const heldByRooms = devices.length === 0 ? new Set() : await (options.held ?? roomHeldCards)()
+  const cards = runnerGpuCards({ devices, perDevice, heldByRooms, busy: options.busy })
+  return {
+    cards,
+    perDevice,
+    slots: cards.reduce((sum, card) => sum + card.capacity, 0),
+    busy: cards.reduce((sum, card) => sum + card.busy, 0),
+    census: heldByRooms !== null,
+  }
+}
+
+/**
+ * A card counted in parts: 840 divides by every runs-per-card the setting
+ * allows (1–8, SETTINGS_LIMITS.gpuPerDevice), so shares taken under different
+ * settings add up exactly, with no rounding to decide whether a card is full.
+ */
+const CARD_PARTS = 840
+
+/** What a run taken at `perDevice` runs per card holds of its card, in CARD_PARTS. */
+function partsOf(perDevice: number): number {
+  const range = SETTINGS_LIMITS.gpuPerDevice
+  return CARD_PARTS / Math.min(range.max, Math.max(range.min, Math.floor(perDevice) || range.min))
+}
+
+/**
+ * How many runs each card holds now, in the share a new run would take there
+ * (its `capacity`, the runs per card in force): what pickRunnerGpu compares
+ * with the capacity. Pure, so the queue can call it inside its transaction.
+ *
+ * Weighed by the share each run was given (QueueRow.gpuPerDevice), not
+ * counted by heads. A run taken at one run per card holds the whole card —
+ * uncapped and unwatched (docker-runner.ts · heldShareMb) — and a run taken
+ * at two holds half of it, whatever the owner has set since: counted by
+ * heads, a raise from one to two put a capped run next to an uncapped one,
+ * and from two to four put 12 + 12 + 6 + 6 GB on a 24 GB card. A run with no
+ * number pinned (a row from before the pin) and a container still awaiting
+ * removal (`held`) count as one share at the number in force, as both did
+ * before. The answer is the places taken: the capacity less the whole
+ * shares still left, so a card is full exactly when a new share no longer
+ * fits — the same answer as a head count while every run has the share in
+ * force.
+ */
+export function cardsInUse(
+  cards: ReadonlyArray<{ id: string; capacity: number }>,
+  runs: ReadonlyArray<{ device: string; perDevice: number | null }>,
+  held: Readonly> = {},
+): Map {
+  const capacity = new Map(cards.map((card) => [card.id, card.capacity]))
+  const parts = new Map()
+  const add = (device: string, value: number) => parts.set(device, (parts.get(device) ?? 0) + value)
+  // A card that takes no run now (a class holds it) has no number in force: what is on it counts whole.
+  const inForce = (device: string) => partsOf(capacity.get(device) || 1)
+  for (const run of runs) add(run.device, run.perDevice === null ? inForce(run.device) : partsOf(run.perDevice))
+  for (const [device, count] of Object.entries(held)) {
+    if (Number.isFinite(count) && count > 0) add(device, Math.floor(count) * inForce(device))
+  }
+  const used = new Map()
+  for (const [device, load] of parts) {
+    const places = capacity.get(device) ?? 0
+    used.set(device, places > 0 ? places - Math.max(0, Math.floor((CARD_PARTS - load) / partsOf(places))) : Math.ceil(load / CARD_PARTS))
+  }
+  return used
+}
+
+/**
+ * How many of the runner cards' places are taken, for the console
+ * (QueueSnapshot.gpu.busy): the places taken on each card that takes runs
+ * (cardsInUse), and one for each run on a card that takes none now — given
+ * back to classes, or held by one — which is still a run.
+ */
+export function placesTaken(
+  cards: ReadonlyArray<{ id: string; capacity: number }>,
+  runs: ReadonlyArray<{ device: string; perDevice: number | null }>,
+): number {
+  const used = cardsInUse(cards, runs)
+  const taking = new Map(cards.filter((card) => card.capacity > 0).map((card) => [card.id, card.capacity]))
+  let taken = runs.filter((run) => !taking.has(run.device)).length
+  for (const [device, capacity] of taking) taken += Math.min(capacity, used.get(device) ?? 0)
+  return taken
+}
+
+/**
+ * Which card the next GPU run takes: the least loaded card that still has
+ * room, the first in the list on a tie (the list is in KERNEL_GPUS order);
+ * `null` — every card is full or held by a class, and the run waits. `busy`
+ * is what each card holds in places (cardsInUse): runs, while every run
+ * there has the share in force.
+ *
+ * Least loaded rather than "the first with room": two runs time-sharing one
+ * card while the second card idles is half the speed for nothing. Pure, so
+ * the queue can call it inside its transaction with the counts it reads there.
+ */
+export function pickRunnerGpu(
+  capacity: ReadonlyArray<{ id: string; capacity: number }>,
+  busy: ReadonlyMap,
+): string | null {
+  let best: { id: string; load: number } | null = null
+  for (const card of capacity) {
+    const load = busy.get(card.id) ?? 0
+    if (load >= card.capacity) continue
+    if (best === null || load < best.load) best = { id: card.id, load }
+  }
+  return best?.id ?? null
+}
+
+/* ---------------------------------------------------------- the owner's form */
+
+/**
+ * The «Видеокарты для посылок» block of the Resources tab: every card of
+ * KERNEL_GPUS with its name, size and holder, the chosen ones, runs per card
+ * and the share that makes. Under the broker an empty, managed block: the
+ * cluster has the cards.
+ */
+export async function gpuSettingsView(options: { waitMs?: number } = {}): Promise {
+  const setting = runnerGpuSetting()
+  const perDevice = gpuPerDeviceSetting()
+  const bounds = { perDevice: { min: SETTINGS_LIMITS.gpuPerDevice.min, max: SETTINGS_LIMITS.gpuPerDevice.max } }
+  const source = { devices: setting.source, perDevice: perDevice.source }
+  if (!runnerCardsHere()) {
+    return { cards: [], devices: [], perDevice: perDevice.value, sliceMb: null, source, bounds, managed: true }
+  }
+  const all = [...new Set(gpuDevices())]
+  const chosen = new Set(setting.devices)
+  const [cards, held, running] = await Promise.all([
+    gpuCards(all, { waitMs: options.waitMs ?? CARD_WAIT_MS }),
+    // For the picture only: an unread census shows every card as nobody's.
+    all.length === 0 ? Promise.resolve([]) : gpuAssignments().catch(() => []),
+    /*
+     * And the submissions' containers, by the same label: a card the owner
+     * gave back to classes while a submission still ran on it read «свободна»
+     * here, while the bar above counted it as the submissions' and a class
+     * starting on it was refused until the run ended.
+     */
+    all.length === 0 ? Promise.resolve([] as string[]) : submissionGpuCards().catch(() => [] as string[]),
+  ])
+  const heldByRoom = new Set(held.map((room) => room.gpu))
+  const heldBySubmission = new Set(running)
+  return {
+    cards: cards.map((card) => ({
+      id: card.id,
+      name: card.name,
+      memoryMb: card.memoryMb,
+      holder: chosen.has(card.id) ? 'runners' : heldByRoom.has(card.id) ? 'rooms' : heldBySubmission.has(card.id) ? 'runners' : 'free',
+      heldByRoom: heldByRoom.has(card.id),
+      heldBySubmission: heldBySubmission.has(card.id),
+    })),
+    devices: all.filter((device) => chosen.has(device)),
+    perDevice: perDevice.value,
+    sliceMb: sliceOverCards(cards.filter((card) => chosen.has(card.id)), perDevice.value),
+    source,
+    bounds,
+    managed: false,
+    fallback: runnerGpuFallback(all),
+  }
+}
+
+/**
+ * What the owner's «вернуть …» falls back to: the same rule as the setting
+ * with nothing saved (settings.ts · resolveRunnerGpus, resolveGpuPerDevice),
+ * so the form can show the cards and the share Save will leave, and name
+ * .env when that is where they come from. Only the cards `all` (KERNEL_GPUS)
+ * lists, as for the chosen ones.
+ */
+function runnerGpuFallback(all: readonly string[]): NonNullable {
+  const devices = resolveRunnerGpus({ saved: null, env: process.env.COMPETITION_GPUS })
+  const perDevice = resolveGpuPerDevice({ saved: null, env: process.env.COMPETITION_GPU_PER_DEVICE })
+  const named = new Set(devices.devices)
+  return {
+    devices: all.filter((device) => named.has(device)),
+    perDevice: perDevice.value,
+    source: { devices: devices.source === 'env' ? 'env' : 'default', perDevice: perDevice.source === 'env' ? 'env' : 'default' },
+  }
+}
diff --git a/server/src/competitions/harness.ts b/server/src/competitions/harness.ts
index eabb9bd4..e2eee775 100644
--- a/server/src/competitions/harness.ts
+++ b/server/src/competitions/harness.ts
@@ -1328,7 +1328,221 @@ else:
     sys.stdout.write(base64.b64encode(read(name)).decode("ascii"))
 `
 
-/** What lies in the harness directory: the file name is what the container calls it. */
+// A submission's share of a shared video card, held from inside its own
+// Python: the soft half of the share (competitions/gpu.ts explains the two
+// halves; the hard one is docker-runner.ts · VramWatch). It lies in a
+// directory of its own, `site/`, because that directory goes on PYTHONPATH:
+// the whole harness there would make run_notebook, colloq_metric and the rest
+// importable by name from the participant's notebook, and shadow whatever of
+// theirs is called the same. The torch cap is the draft tried on the lab VM on
+// 13 Oct 2026 (2 × RTX 4090): with a 4096 MB share the 17th allocation of 256
+// MiB raised torch.OutOfMemoryError, and without the variable nothing changed.
+// The same day's review found two holes in it: the finder left at the first
+// question about torch, so a library that only asked (polars, at its import)
+// took the cap away; and CatBoost's GPU pool, sized as a part of the card and
+// never seen by torch, would take nearly all of it. Both are checked against
+// stand-in packages under a real python (tests/competition-gpu-runs.test.mts).
+const SITECUSTOMIZE = `"""Colloq: a submission's share of a shared video card, held inside its own Python.
+
+Several submissions may time-share one card, and each is promised a share of
+its video memory. The runner then passes COLLOQ_GPU_MEMORY_MB (the share less
+what the CUDA context takes outside any allocator) and COLLOQ_GPU_CARD_MB (the
+card the share was cut from), and puts this directory on PYTHONPATH: Python
+imports sitecustomize in every process it starts, the notebook's kernel
+included, before a line of the participant's code runs.
+
+Nothing is imported here. Importing torch costs a second and a half and half a
+gigabyte of memory, and most processes of the container never use it (the
+harness itself does not). A finder waits in front of the import system
+instead, for the libraries of the GPU images that would take more than the
+share on their own:
+
+torch: when the notebook imports it, a callback is queued for the moment CUDA
+initialises, and the callback caps torch's caching allocator at
+COLLOQ_GPU_MEMORY_MB on device 0 (inside the container the card given to it is
+the only one). An allocation past the cap then raises torch.OutOfMemoryError
+in the participant's own cell, with the numbers in its message, instead of
+taking the memory of the other runs on the card.
+
+catboost: a GPU fit takes gpu_ram_part of the card's free memory at once, 95
+per cent by default, as one pool torch never sees; on a card shared by two
+runs that is nearly the whole card, and the watchdog stops the run for it
+however small its data. Unless the participant names their own, gpu_ram_part
+becomes COLLOQ_GPU_MEMORY_MB over COLLOQ_GPU_CARD_MB: the free memory is never
+more than the card, so the pool never outgrows the share, whatever the
+neighbours hold (with them holding much, it is smaller, and a participant who
+knows better names their own). It is set
+where catboost 1.2 hands its parameters on to the GPU, the training
+parameters of fit, grid and randomized search and feature selection
+(CatBoost._prepare_train_params) and those of cv (catboost.core._cv), each a
+copy made for that call; the model's own parameters stay as written.
+
+xgboost needs nothing: on device="cuda" it allocates as it goes, through a
+caching allocator that grows on demand, never a part of the card up front.
+What outgrows the share there meets the watchdog, like anything else outside
+torch's cap.
+
+The finder stays in front for the whole process, and a library is hooked when
+its module is executed, not when someone asks about it. Libraries ask whether
+torch is installed long before anyone imports it (polars does, at its own
+import, through importlib.util.find_spec), and the first version of this
+finder, which left at the first question, was gone by the real import: a
+notebook that imported polars first ran without a cap (13 Oct 2026). So every
+answer it gives carries the hook, and the hook runs once per process, on the
+first execution of the library's module, whichever answer that came from.
+
+This is the soft half, and it lives in the participant's process: code that
+lifts the cap, or a library that allocates past it, meets the hard half, the
+runner's watchdog, which reads the card from outside and stops a run that
+holds more than its share. Without the variables (a whole card, a card of
+unknown size, a run on the CPU) nothing here does anything, and whatever goes
+wrong here is said on stderr and never breaks the participant's import.
+"""
+import _thread
+import functools
+import importlib.util
+import os
+import sys
+
+
+def _mb(name):
+    try:
+        return int(os.environ.get(name) or 0)
+    except ValueError:
+        return 0
+
+
+def _not_applied(what, error):
+    print(f"colloq: {what} not applied: {error}", file=sys.stderr)
+
+
+def _cap_torch(torch):
+    mb = _mb("COLLOQ_GPU_MEMORY_MB")
+    if mb <= 0:
+        return
+
+    def apply():
+        try:
+            total = torch.cuda.get_device_properties(0).total_memory
+            torch.cuda.set_per_process_memory_fraction(min(1.0, mb * 2**20 / total), 0)
+        except Exception as e:  # never break the participant's CUDA
+            _not_applied("GPU memory cap", e)
+
+    try:
+        # Queued, not called: the fraction needs an initialised device, and
+        # initialising CUDA here would take the card's memory on import.
+        torch.cuda._lazy_call(apply)
+    except Exception as e:  # never break the participant's import
+        _not_applied("GPU memory cap", e)
+
+
+def _catboost_part():
+    card = _mb("COLLOQ_GPU_CARD_MB")
+    part = _mb("COLLOQ_GPU_MEMORY_MB") / card if card > 0 else 0
+    return part if 0 < part <= 1 else None
+
+
+def _default_catboost(catboost):
+    part = _catboost_part()
+    if part is None:
+        return
+
+    def default(params):
+        # A GPU fit, and only when the participant named no part of their own.
+        if str(params.get("task_type") or "").upper() == "GPU" and params.get("gpu_ram_part") is None:
+            params["gpu_ram_part"] = part
+
+    try:
+        # The two doors of catboost 1.2; a version without them loads as it is.
+        core = catboost.core
+        prepare = core.CatBoost._prepare_train_params
+        cv = core._cv
+    except Exception as e:
+        _not_applied("CatBoost GPU memory default", e)
+        return
+
+    @functools.wraps(prepare)
+    def _prepare_train_params(self, *args, **kwargs):
+        prepared = prepare(self, *args, **kwargs)
+        try:
+            default(prepared["params"])
+        except Exception as e:  # never break the participant's fit
+            _not_applied("CatBoost GPU memory default", e)
+        return prepared
+
+    @functools.wraps(cv)
+    def _cv(params, *args, **kwargs):
+        try:
+            default(params)
+        except Exception as e:
+            _not_applied("CatBoost GPU memory default", e)
+        return cv(params, *args, **kwargs)
+
+    core.CatBoost._prepare_train_params = _prepare_train_params
+    core._cv = _cv
+
+
+class _GpuFinder:
+    """Finds nothing itself: the finders behind it find a library, and it hooks the library's execution."""
+
+    def __init__(self, hooks):
+        self._hooks = hooks
+        self._done = set()
+        # Threads inside the lookup below, which asks the import system again
+        # and so this finder again: there it must step aside, not leave.
+        self._asking = set()
+
+    def find_spec(self, name, path=None, target=None):
+        hook = self._hooks.get(name)
+        if hook is None or name in self._done or name in sys.modules:
+            return None
+        asker = _thread.get_ident()
+        if asker in self._asking:
+            return None
+        self._asking.add(asker)
+        try:
+            spec = importlib.util.find_spec(name)
+        except Exception:
+            return None  # the import system asks the others itself and says what they say
+        finally:
+            self._asking.discard(asker)
+        loader = getattr(spec, "loader", None)
+        run = getattr(loader, "exec_module", None)
+        if run is None:
+            return spec
+
+        def exec_module(module):
+            run(module)
+            # Once per process, and on the library itself (a loader may serve several modules).
+            if getattr(module, "__name__", None) == name and name not in self._done:
+                self._done.add(name)
+                try:
+                    hook(module)
+                except Exception as e:  # never break the participant's import
+                    _not_applied(f"{name} hook", e)
+
+        try:
+            # The path finders make a new loader for every answer, so this
+            # hooks the answer given now, whoever asked, and nothing else.
+            loader.exec_module = exec_module
+        except Exception as e:
+            _not_applied(f"{name} hook", e)
+        return spec
+
+
+_HOOKS = {}
+if _mb("COLLOQ_GPU_MEMORY_MB") > 0:
+    _HOOKS["torch"] = _cap_torch
+    if _catboost_part() is not None:
+        _HOOKS["catboost"] = _default_catboost
+if _HOOKS:
+    sys.meta_path.insert(0, _GpuFinder(_HOOKS))
+`
+
+/**
+ * What lies in the harness directory: the file name is what the container
+ * calls it, under /harness — a subdirectory included (`site/`).
+ */
 const FILES: ReadonlyArray = [
   ['run_notebook.py', RUN_NOTEBOOK],
   ['kernel_streams.py', KERNEL_STREAMS],
@@ -1349,8 +1563,15 @@ os.environ['COMP_PUBLIC_PERCENT'] = str(config['publicPercent'])
 os.environ['COMP_SPLIT_SEED'] = config['splitSeed']
 runpy.run_path('/harness/score_metric.py', run_name='__main__')
 `],
+  ['site/sitecustomize.py', SITECUSTOMIZE],
 ]
 
+/**
+ * The directory the runner puts on PYTHONPATH for a run with a share of a
+ * card, as the container sees it (the harness is mounted at /harness).
+ */
+export const HARNESS_SITE = '/harness/site'
+
 /**
  * The source of the split module — for the suite that checks it against
  * `@shared/competitions`.
@@ -1388,6 +1609,8 @@ export function harnessDir(): string {
     // Already there, so we do not rewrite it: in a directory with a hash in its
     // name the content is either the same or not there at all.
     if (competitionsFs.existsSync(file)) continue
+    // A file in a subdirectory (`site/`) needs the subdirectory first.
+    competitionsFs.mkdirSync(path.dirname(file), { recursive: true })
     competitionsFs.writeFileSync(file, Buffer.from(body, 'utf8'), { mode: 0o644 })
   }
   materialized = dir
@@ -1401,9 +1624,13 @@ export function brokerHarnessDir(): string {
   competitionsFs.chmodSync(dir,0o755)
   for(const [name,body] of FILES) {
     const file=path.join(dir,name)
+    // A file in a subdirectory (`site/`) gets the subdirectory, readable by the Pod's user like the root.
+    const folder=path.dirname(file)
+    if(folder!==dir){competitionsFs.mkdirSync(folder,{recursive:true});competitionsFs.chmodSync(folder,0o755)}
     const hash=createHash('sha256').update(body).digest('hex')
     if(competitionsFs.existsSync(file) && createHash('sha256').update(competitionsFs.readFileSync(file) as Buffer).digest('hex')===hash)continue
-    const temp=path.join(dir,`.${name}-${HARNESS_REVISION}.tmp`)
+    // The temporary file next to its target: a rename never crosses directories.
+    const temp=path.join(folder,`.${path.basename(file)}-${HARNESS_REVISION}.tmp`)
     competitionsFs.writeFileSync(temp,Buffer.from(body,'utf8'),{mode:0o644})
     competitionsFs.renameSync(temp,file)
   }
diff --git a/server/src/competitions/panel.ts b/server/src/competitions/panel.ts
index a10adcb2..32de8075 100644
--- a/server/src/competitions/panel.ts
+++ b/server/src/competitions/panel.ts
@@ -96,10 +96,22 @@ export const MIN_REMAINING_MS = 5_000
 export interface EtaInput {
   /** How many submissions run at once. */
   slots: number
-  /** Jobs running now: how long each is expected to take and how long it has run. */
-  running: readonly { expectedMs: number; elapsedMs: number }[]
-  /** Waiting jobs in the order the runner will take them, each with its expected run time. */
-  waiting: readonly { expectedMs: number }[]
+  /**
+   * Jobs running now: how long each is expected to take and how long it has
+   * run, and whether it holds a share of a runner card.
+   */
+  running: readonly { expectedMs: number; elapsedMs: number; gpu?: boolean }[]
+  /**
+   * Waiting jobs in the order the runner prefers them (fairOrder), each with
+   * its expected run time and whether it needs a card share.
+   */
+  waiting: readonly { expectedMs: number; gpu?: boolean }[]
+  /**
+   * How many runs may hold a card share at once (QueueSnapshot.gpu.slots).
+   * Absent — no job waits for a card here: the cluster places GPU work
+   * itself, and it is simulated like any other.
+   */
+  gpuSlots?: number
 }
 
 export interface Eta {
@@ -114,29 +126,78 @@ export interface Eta {
  * not until its start: the person waits for a number, not for a container.
  *
  * Computed by simulating the slots, not by multiplying the position by the
- * average: a min-heap of the moments slots free up (a running job frees one
- * after what it still has left, never sooner than five seconds; an idle slot
- * is free now), and the waiting jobs take them in queue order. With two
- * executors the fifth in the queue waits half as long, and a number computed
- * without this lies by exactly that factor.
+ * average: the moments slots free up (a running job frees one after what it
+ * still has left, never sooner than five seconds; an idle slot is free now),
+ * and the waiting jobs take them in queue order. With two executors the
+ * fifth in the queue waits half as long, and a number computed without this
+ * lies by exactly that factor.
  *
  * More running jobs than slots (the owner lowered the number mid-class): the
  * first of them to finish free nothing for the queue — only the last `slots`
  * of them do.
+ *
+ * With `gpuSlots`, the card shares are simulated as the queue takes them
+ * (store.ts · selectNext): a job that needs a card takes a slot AND a share,
+ * and while no share is free it is passed over, and the jobs behind it take
+ * the slot. So a GPU job may start after jobs that come after it in the
+ * order, and the caller numbers places by start (forecast.ts). `null` — a job
+ * that needs a card where there are no shares at all: it does not start, and
+ * nobody can say when it will.
  */
-export function queueEtas(input: EtaInput): Eta[] {
+export function queueEtas(input: EtaInput): Array {
   const slots = Math.max(1, Math.floor(input.slots))
-  const left = input.running
-    .map((job) => Math.max(job.expectedMs - job.elapsedMs, MIN_REMAINING_MS))
-    .sort((a, b) => a - b)
-  const free = new MinHeap(left.slice(Math.max(0, left.length - slots)))
-  while (free.size < slots) free.push(0)
-  return input.waiting.map((job) => {
-    const startMs = free.pop()
-    const etaMs = startMs + Math.max(0, job.expectedMs)
-    free.push(etaMs)
-    return { startMs, etaMs }
-  })
+  const cards = input.gpuSlots === undefined ? null : Math.max(0, Math.floor(input.gpuSlots))
+  const needsCard = (job: { gpu?: boolean }): boolean => cards !== null && job.gpu === true
+  // When the jobs holding a slot end; those holding a card share apart, since they free both.
+  const slotEnds = new MinHeap()
+  const cardEnds = new MinHeap()
+  let freeSlots = slots - input.running.length
+  let freeCards = cards ?? 0
+  for (const job of input.running) {
+    const left = Math.max(job.expectedMs - job.elapsedMs, MIN_REMAINING_MS)
+    if (needsCard(job)) {
+      cardEnds.push(left)
+      freeCards--
+    } else slotEnds.push(left)
+  }
+  const etas: Array = input.waiting.map(() => null)
+  const pending = input.waiting.map((_, index) => index)
+  let now = 0
+  for (;;) {
+    // Everything that can start now starts, in queue order, the way one pass of the pump takes it.
+    for (let at = 0; at < pending.length && freeSlots > 0; ) {
+      const index = pending[at]
+      const job = input.waiting[index]
+      const card = needsCard(job)
+      if (card && freeCards <= 0) {
+        at++
+        continue
+      }
+      const etaMs = now + Math.max(0, job.expectedMs)
+      etas[index] = { startMs: now, etaMs }
+      freeSlots--
+      if (card) {
+        freeCards--
+        cardEnds.push(etaMs)
+      } else slotEnds.push(etaMs)
+      pending.splice(at, 1)
+    }
+    if (pending.length === 0) break
+    // Then time moves to the next end; every job that ends at that moment frees its slot first.
+    const next = Math.min(slotEnds.size > 0 ? slotEnds.peek() : Infinity, cardEnds.size > 0 ? cardEnds.peek() : Infinity)
+    if (next === Infinity) break
+    now = next
+    while (slotEnds.size > 0 && slotEnds.peek() === now) {
+      slotEnds.pop()
+      freeSlots++
+    }
+    while (cardEnds.size > 0 && cardEnds.peek() === now) {
+      cardEnds.pop()
+      freeSlots++
+      freeCards++
+    }
+  }
+  return etas
 }
 
 /**
@@ -170,6 +231,11 @@ class MinHeap {
     return this.items.length
   }
 
+  /** The smallest, left where it is; only for a heap that is not empty. */
+  peek(): number {
+    return this.items[0]
+  }
+
   push(value: number): void {
     const items = this.items
     items.push(value)
@@ -299,6 +365,8 @@ export type InputRefusal =
   | { field: 'environment'; why: 'chars' }
   | { field: string; why: 'range'; min: number; max: number }
   | { field: string; why: 'value' }
+  /** A video card for an environment that runs on the CPU (`gpuEnvironmentRefusal`). */
+  | { field: 'limits.gpu'; why: 'cpuEnvironment'; environment: string }
 
 const CHOICES = {
   direction: ['lower', 'higher'],
@@ -380,6 +448,11 @@ export function parseCompetitionInput(
       }
       out[name] = value
     }
+    // A switch, like late intake below: true or false, never "anything truthy".
+    if (Object.prototype.hasOwnProperty.call(limits, 'gpu')) {
+      if (typeof limits.gpu !== 'boolean') return { refusal: { field: 'limits.gpu', why: 'value' } }
+      out.gpu = limits.gpu
+    }
     input.limits = out
   }
 
@@ -452,6 +525,35 @@ export function parseCompetitionInput(
   return { input }
 }
 
+/**
+ * A video card is for an environment that needs one; `null` — the edit is fine.
+ *
+ * Judged on what the competition is AFTER the edit — the environment and the
+ * switch as sent, or as they are where the form did not send them — so both
+ * ways in are closed: turning the card on for a CPU environment, and moving a
+ * competition that has a card to a CPU environment. A card under a CPU
+ * environment is a card held for nothing: no package there can use it, and
+ * the run takes it from a submission that could. Only an edit that touches
+ * one of the two is judged: a title saved under an environment that has since
+ * lost its GPU directive is not the place to stop someone.
+ *
+ * `needsGpu` comes in from outside (competitions/gpu.ts · environmentNeedsGpu):
+ * whether an environment needs a GPU is read from its package list on disk,
+ * or from the cluster's catalog, and this module reads neither.
+ */
+export function gpuEnvironmentRefusal(
+  input: CompetitionInput,
+  current: { environment: string; limits: { gpu?: boolean } } | null,
+  needsGpu: (environment: string) => boolean,
+): InputRefusal | null {
+  if (input.environment === undefined && input.limits?.gpu === undefined) return null
+  // A new competition is born on `base` (store.ts · createCompetition).
+  const environment = input.environment ?? current?.environment ?? 'base'
+  const gpu = input.limits?.gpu ?? current?.limits.gpu ?? false
+  if (!gpu || needsGpu(environment)) return null
+  return { field: 'limits.gpu', why: 'cpuEnvironment', environment }
+}
+
 /**
  * An integer from the request body; `null` means it is not a number
  * (including `"5"` with garbage).
@@ -485,6 +587,12 @@ export interface QueueOrderRow {
  * late work; whoever already has something running lets the others go ahead;
  * one's own k-th submission gets in line behind everyone else's (k−1)-th; on
  * a tie, whoever sent it first.
+ *
+ * The one part of `selectNext` that is not an order is not here: work that
+ * needs a card share is passed over while none is free. Whether it is
+ * depends on the moment, so that part lives in the simulation (queueEtas),
+ * and the places on both screens are numbered by the simulated starts
+ * (forecast.ts) — with no GPU work waiting, exactly this order.
  */
 export function fairOrder(
   waiting: readonly T[],
diff --git a/server/src/competitions/runner-port.ts b/server/src/competitions/runner-port.ts
index d29363b5..e5cb8057 100644
--- a/server/src/competitions/runner-port.ts
+++ b/server/src/competitions/runner-port.ts
@@ -110,10 +110,39 @@ export interface RunRequest {
   /** Host-only destination for bounded artifacts exported from the attempt. */
   resultDir: string
   limits: StepLimits
+  /**
+   * The runner card this attempt holds, given by the queue when it took the
+   * job (competitions/gpu.ts · pickRunnerGpu); absent — the notebook runs on
+   * the CPU, exactly as before cards existed.
+   *
+   * `device` is the `KERNEL_GPUS` entry as docker names it; inside the
+   * container the card is index 0. `sliceMb` — the video memory this run is
+   * promised (the card's memory over `perDevice`), `null` when the card did
+   * not say its size: then nothing is capped and nothing is watched, because
+   * a limit nobody can state is not one to enforce. `perDevice` — how many
+   * runs share the card; with one the run has the whole card and needs no
+   * cap at all. Only the notebook step ever gets this: the metric step holds
+   * the hidden answers.
+   *
+   * Under Kubernetes the queue holds no card (competition_queue.gpu stays
+   * NULL): the cluster schedules the Pod onto a unit of its own GPU resource,
+   * and a node without one sends the job back as a `gpu` resource deferral.
+   * A GPU notebook there carries `CLUSTER_GPU` below — an empty device, "the
+   * cluster picks", no share of ours and nothing to watch — and the broker
+   * runner reads only that the field is there (`gpu: true` in its intent).
+   */
+  gpu?: { device: string; sliceMb: number | null; perDevice: number }
   /** Called as the run goes; the queue puts this into the submission row. */
   onProgress?: (progress: RunProgress) => void
 }
 
+/**
+ * A GPU notebook under Kubernetes (RunRequest.gpu): no device of ours, no
+ * share we could state, and so no cap and no watchdog — the cluster's device
+ * plugin decides what a unit of its GPU resource is.
+ */
+export const CLUSTER_GPU: Readonly> = Object.freeze({ device: '', sliceMb: null, perDevice: 1 })
+
 /** What became of the notebook. The same on both runners. */
 export interface RunOutcome {
   /** The harness's word, untranslated and unembellished (`RunVerdict`). */
@@ -156,10 +185,29 @@ export interface RunDiagnostics {
   exit: number | null
   oomKilled: boolean
   backend: CompetitionBackend
-  /** Who cut the container off from outside: time limit, printing, disk writes. */
-  killedBy?: 'wall' | 'output' | 'disk'
+  /**
+   * Who cut the container off from outside: time limit, printing, disk
+   * writes, video memory past the run's share of a card (`gpu`: two
+   * samples in a row above the share plus its tolerance, competitions/gpu.ts ·
+   * GPU_TOLERANCE_MB), or a shared card that could not be read at all
+   * (`gpu-unread`: docker-runner.ts · VRAM_UNREAD_LIMIT samples in a row
+   * failed, and the run was stopped rather than left unwatched).
+   */
+  killedBy?: 'wall' | 'output' | 'disk' | 'gpu' | 'gpu-unread'
   /** Memory peak read by the harness from inside — it shows "right at the limit". */
   peakBytes?: number | null
+  /**
+   * The most video memory the run's container was seen holding, MB, for the
+   * teacher; `null` — no sample answered; absent — a run nobody watched: no
+   * card, a whole card (nothing to enforce), or a share of unknown size.
+   */
+  gpuPeakMb?: number | null
+  /**
+   * The last samples that answered, MB, oldest first, at most four — for a
+   * run killed for its card, the two that decided it (docker-runner.ts ·
+   * gpuOverShare). Absent, like the peak, for a run nobody watched.
+   */
+  gpuSamplesMb?: number[]
 }
 
 /** An order to compute the metric. The participant is not in this container. */
@@ -214,6 +262,13 @@ export interface Capacity {
   memAvailableMb?: number | null
   /** Memory still held by containers whose removal failed and is being retried. */
   heldMb?: number
+  /**
+   * Runner cards still held the same way, card → containers: a notebook
+   * container that outlived its attempt keeps its share of the card until it
+   * is gone, and the queue counts it as a run on that card (store.ts ·
+   * takeNext). Absent — none, or a runner without cards of its own.
+   */
+  heldGpus?: Record
 }
 
 export interface CompetitionRunner {
diff --git a/server/src/competitions/runner.ts b/server/src/competitions/runner.ts
index db0339b4..65b91cad 100644
--- a/server/src/competitions/runner.ts
+++ b/server/src/competitions/runner.ts
@@ -31,7 +31,8 @@ import { competitionCapabilities } from './capabilities.js'
 import { admitJob, MAX_SLOTS, slotsResolution } from './capacity.js'
 import { saveSlots } from './settings.js'
 import { submissionUsesCurrentBase, type AttemptProvenance } from './provenance.js'
-import { tr } from '@shared/i18n'
+import { gpuKillAboveMb, gpuSliceFor, noteClusterGpuSlots, runnerCardsHere, runnerGpuDevices, runnerGpuPerDevice, runnerGpuState } from './gpu.js'
+import { formatNumber, tr } from '@shared/i18n'
 /*
  * Not decoration: `admin/settings` installs the translator that reads the
  * instance language, and the strings the runner writes to the participant go
@@ -45,6 +46,7 @@ import { dependencyMessage } from '../dependencies/messages.js'
 import { dependencySize, packagesFit, packagesRoomMb } from '@shared/dependencies'
 import {
   BOOT,
+  cardWorkWaiting,
   enqueue,
   deferResourceAttempt,
   discardUnstartedRun,
@@ -59,6 +61,7 @@ import {
   openFileBytes,
   sealedFileBytes,
   noteQueueContainer,
+  noteQueueGpuShare,
   nextQueueRow,
   orphanedRuns,
   queuePaused,
@@ -71,6 +74,7 @@ import {
   takeNext,
   updateSubmission,
   waitingCount,
+  type QueueGpu,
   type QueueRow,
 } from './store.js'
 import {
@@ -90,16 +94,19 @@ import {
   SOLUTION_FILE,
   SUBMISSION_FILE,
 } from './storage.js'
-import { METRIC_NAME, competitionDockerDiagnostics } from './docker-runner.js'
+import { METRIC_NAME, VRAM_UNREAD_LIMIT, competitionDockerDiagnostics } from './docker-runner.js'
 import { startSubmissionSweep, stopSubmissionSweep } from './housekeeping.js'
 import { CompetitionResourcePending } from './broker-runner.js'
 import {
+  CLUSTER_GPU,
   competitionBackend,
   competitionRunner,
   limitsFor,
+  type Capacity,
   type CompetitionBackend,
   type DependencyFailure,
   type RunOutcome,
+  type RunRequest,
   type ScoreOutcome,
 } from './runner-port.js'
 import {
@@ -254,6 +261,13 @@ let timer: NodeJS.Timeout | null = null
 let pumping = false
 let stopped = false
 
+/**
+ * Until when no GPU notebook is taken under the cluster (queueCards): the
+ * backoff of the last one the cluster sent back for want of a unit. Zero —
+ * no rest.
+ */
+let clusterGpuRestUntil = 0
+
 /**
  * Take as much work as the machine and the setting allow, and do NOT wait for
  * it.
@@ -271,7 +285,8 @@ export async function pumpOnce(): Promise {
     const slots = queueSlots()
     if (runningRows().length >= slots) return 0
     if (waitingCount() === 0) return 0
-    if (!(await competitionCapabilities()).execution.available) return 0
+    const capabilities = await competitionCapabilities()
+    if (!capabilities.execution.available) return 0
     const capacity = await competitionRunner().capacity()
     const usableMb = capacity.usableMb !== undefined ? capacity.usableMb : capacity.availableMb
     const heldMb = capacity.heldMb ?? 0
@@ -284,9 +299,17 @@ export async function pumpOnce(): Promise {
     let admittedMb = 0
     let availableDiskBytes: number | null = null
     try { const disk = fs.statfsSync(competitionsDir); availableDiskBytes = disk.bavail * disk.bsize } catch { /* unknown capacity */ }
+    /*
+     * The runner cards, once per pass: what each may hold is a census of the
+     * rooms' containers, and that is a docker call. Who holds them is not read
+     * here: the queue counts its own running rows at every take, so the
+     * second take of this pass already sees the card the first one took.
+     */
+    const gpu = await queueCards(capacity, noteClusterGpuSlots(capabilities.gpuSlots))
     for (;;) {
       if (runningRows().length >= slots) break
-      const candidate = nextQueueRow()
+      // The same filter as the take below: the work admitted is the work taken.
+      const candidate = nextQueueRow(Date.now(), gpu)
       if (!candidate) break
       const competition = getCompetition(candidate.competitionId)
       const needMb = competition ? runReservationMb(competition, candidate.kind) : RUN_RESERVE_MB
@@ -316,7 +339,7 @@ export async function pumpOnce(): Promise {
         break
       }
       forgetWarning()
-      const row = takeNext({ boot: BOOT, slots })
+      const row = takeNext({ boot: BOOT, slots, gpu })
       if (!row) { release(); break }
       admittedMb += needMb
       started++
@@ -340,6 +363,49 @@ export async function pumpOnce(): Promise {
   return started
 }
 
+/**
+ * The runner cards for one pass of the pump (store.ts · QueueGpu).
+ *
+ * A GPU notebook needs a general slot AND a free share of a card, and the
+ * queue passes it over while no share is free instead of waiting behind it:
+ * CPU work flows past a GPU notebook that waits for a card, and the GPU
+ * notebook waits for nothing else but a general slot (store.ts ·
+ * selectNext). The cards are this machine's — the docker runner and its test
+ * stand-in; each may hold `perDevice` runs, none while a class still holds it
+ * or while docker cannot say who does (competitions/gpu.ts ·
+ * runnerGpuState), what it holds already is weighed by the shares its runs
+ * were taken at (gpu.ts · cardsInUse), and a container whose removal failed
+ * still holds its share (Capacity.heldGpus).
+ *
+ * Under the cluster the queue keeps no card (Kubernetes schedules the Pod
+ * onto a unit of its GPU resource, and a node without one sends the job back
+ * as a `gpu` resource deferral), but it does keep to the number of GPU
+ * notebooks the broker has units for (`clusterSlots`, gpu.ts ·
+ * clusterGpuSlots): a GPU notebook taken past it held a general slot for the
+ * half minute its Pod waited, came back to the head of the queue, and was
+ * taken again — with 25 GPU notebooks waiting no CPU work got a slot for
+ * hours. And once the cluster has sent one back for want of a unit, none is
+ * taken until that one's backoff ends (or a GPU notebook of ours ends and
+ * frees a unit): the cluster has said it has none now, and the CPU work goes
+ * past in the meantime, as it does past a full card here.
+ *
+ * Without GPU work waiting there is no census at all: no cards to count, and
+ * a GPU notebook that arrives during the pass simply waits for the next one,
+ * a second away.
+ */
+async function queueCards(capacity: Capacity, clusterSlots: number): Promise {
+  if (!runnerCardsHere()) return Date.now() < clusterGpuRestUntil ? { cards: [] } : { cards: [], cluster: clusterSlots }
+  if (!cardWorkWaiting()) return { cards: [] }
+  try {
+    const state = await runnerGpuState()
+    return { cards: state.cards, held: capacity.heldGpus ?? {} }
+  } catch (error) {
+    // No count, no card: the GPU work waits for the next pass, the rest goes on.
+    warnOnce(`[competitions] runner cards could not be counted: ${error instanceof Error ? error.message : String(error)}`)
+    return { cards: [] }
+  }
+}
+
 /** Wait for everything running in this process. */
 export function settleCompetitionWork(): Promise {
   return Promise.all([...inFlight.values()]).then(() => undefined)
@@ -486,12 +552,16 @@ async function runJob(row: QueueRow): Promise {
     secrets = snapshotSecrets(competition, submission, row).then(() => null, (error: unknown) => error ?? new Error('Secret snapshot failed'))
     if (row.kind === 'metric') await scoreOnly(competition, submission, row, provenance, secrets)
     else await runNotebookThenScore(competition, submission, row, provenance, secrets)
+    // A GPU notebook of ours ended under the cluster: its unit is free, and the next one may ask for it.
+    if (row.kind === 'notebook' && competition.limits.gpu === true && !runnerCardsHere()) clusterGpuRestUntil = 0
   } catch (err) {
     if (err instanceof CompetitionResourcePending && ownsQueueAttempt(row)) {
       const reason=tr('competitions.runtime.resourcesWaiting')
-      const after=deferResourceAttempt(row,reason,Date.now(),`Competition Pod unschedulable (${err.resource}); retrying automatically.`,err.kind)
+      const after=deferResourceAttempt(row,reason,Date.now(),err.teacherReason,err.kind)
       if (after!==null) {
         updateSubmission(submission.id,{cellsDone:submission.cellsDone,cellsTotal:submission.cellsTotal,durationMs:submission.durationMs})
+        // The cluster has no unit now: no GPU notebook asks again until this one may (queueCards).
+        if (err.resource === 'gpu' && !runnerCardsHere()) clusterGpuRestUntil = Math.max(clusterGpuRestUntil, after)
         return
       }
     }
@@ -593,12 +663,21 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
       }
     } catch (error) { preflightError = error }
   }
+  // Read before the cancellation check, like the set's hash: a card nobody
+  // has asked about yet takes a few seconds to say its size.
+  const gpu = unfit || preflightError ? undefined : await runGpu(row, competition)
   // Hashing a large set yields to cancellation before any container exists.
   if (!ownsQueueAttempt(row) || takeCancellation(submission.id, 0)) {
     finishRun(run.id, { finishedAt: Date.now(), verdict: 'exit', cellsDone: 0, cellsTotal: 0,
       teacherError: 'Cancelled before container start' })
     return
   }
+  // After the cancellation, which outranks it: a cancelled run is not put back in the queue.
+  const withdrawn = takenBack(gpu)
+  if (withdrawn !== null) {
+    discardUnstartedRun(run.id, row.attemptId!)
+    throw new CardWithdrawn(withdrawn)
+  }
   let outcome: RunOutcome
   try { outcome = unfit ? {
     status: 'dependency_error', cell: -1, cells: 0, wall: 0, submission: null,
@@ -624,6 +703,7 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
     inputDir: input,
     resultDir: result,
     limits,
+    ...(gpu ? { gpu } : {}),
     onProgress: (progress) => {
       if (!ownsQueueAttempt(row)) return
       const seen = codeCells === null ? progress : clampCells(progress, codeCells)
@@ -637,14 +717,19 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
     if(error instanceof CompetitionResourcePending)discardUnstartedRun(run.id,row.attemptId!)
     throw error
   }
+  const endedAt = Date.now()
+  // Said at once, before anything below waits: a neighbour on the card may be waiting to learn it (neighbourOverran).
+  const stoppedBy = outcome.diagnostics.killedBy
+  if ((stoppedBy === 'gpu' || stoppedBy === 'gpu-unread') && gpu?.device) noteShareKill(gpu.device, endedAt, stoppedBy)
   if (codeCells !== null) outcome = { ...clampCells(outcome, codeCells), diagnostics: { ...outcome.diagnostics, peakBytes: null } }
   // One text for the run row and the submission row. Where packages come from
   // is read only for a failed cell: that is the one outcome a missing module
   // explains.
   const participantError = notebookNote(outcome, competition,
-    outcome.status === 'cell_error' ? packageSources(competition, submission, binding) : null)
+    outcome.status === 'cell_error' ? packageSources(competition, submission, binding) : null, gpu)
   const errorType = outcome.status === 'cell_error' && outcome.diagnostics.killedBy !== 'output' ? errorTypeOf(outcome.detail) : null
-  const briefError = sealed ? briefNote(outcome, competition, errorType) : null
+  const briefError = sealed ? briefNote(outcome, competition, errorType, gpu) : null
+  const teacherError = teacherNote(outcome, gpu)
   finishRun(run.id, {
     finishedAt: Date.now(),
     verdict: outcome.status,
@@ -653,10 +738,29 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
     cellsDone: outcome.cell + 1,
     cellsTotal: outcome.cells,
     participantError,
-    teacherError: teacherNote(outcome),
+    teacherError,
   })
 
   if (!ownsQueueAttempt(row) || takeCancellation(submission.id, outcome.wall)) return
+  /*
+   * Torch ran out of video memory on a shared card while the watchdog was
+   * stopping a neighbour on the same card for going past its share: the
+   * neighbour's burst took the memory this run was promised, in the seconds
+   * the watchdog needs for its two samples. That is not this notebook's
+   * failure, and judging it one told the participant to shrink a model that
+   * fit. So the run goes back to the queue, once (a second time it is judged
+   * like any other), with our own words, and counts toward nothing.
+   */
+  if (gpu?.device && gpu.perDevice > 1 && errorType === 'OutOfMemoryError' && row.resourceRetries === 0
+    && await neighbourOverran(row, gpu.device, endedAt)) {
+    if (!ownsQueueAttempt(row) || takeCancellation(submission.id, outcome.wall)) return
+    const retried = deferResourceAttempt(row, tr('competitions.answer.gpuNeighbourRetry'), Date.now(),
+      tr('competitions.answer.gpuNeighbourRetryTeacher', { device: gpu.device }))
+    if (retried !== null) {
+      updateSubmission(submission.id, { cellsDone: submission.cellsDone, cellsTotal: submission.cellsTotal, durationMs: submission.durationMs })
+      return
+    }
+  }
   publishAttemptArtifacts(competition.id, submission.id, result)
   if (outcome.status !== 'ok') {
     updateSubmission(submission.id, {
@@ -668,7 +772,7 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
       participantError,
       briefError,
       errorType,
-      teacherError: teacherNote(outcome),
+      teacherError,
     })
     return
   }
@@ -701,6 +805,167 @@ async function runNotebookThenScore(competition: Competition, submission: Submis
   await scoreStep(competition, submission, path.join(resultDir(competition.id, submission.id), SUBMISSION_FILE), outcome.wall, outcome, row, provenance, secrets, sealed)
 }
 
+/**
+ * The card a notebook run is given (RunRequest.gpu); `undefined` — the CPU.
+ *
+ * On this machine's cards it is the one the queue took for the attempt
+ * (QueueRow.gpu), with the share promised on THAT card: the panel promises
+ * the smallest card's share, and a run on a bigger card gets its bigger one.
+ * The share is the one the row was taken at (QueueRow.gpuPerDevice), never
+ * the number in force: the queue admitted the run by it, and an owner who
+ * changed «посылок на карту» a second after the take must not hand this run
+ * a share its card has no room for, or a whole uncapped card next to capped
+ * neighbours. What it comes to in MB goes back into the row for the console.
+ *
+ * Under the cluster the queue holds no card, and a GPU notebook carries
+ * CLUSTER_GPU: the Pod asks the cluster for a unit of its GPU resource. A
+ * rescore never gets here with one, and the metric step after the notebook
+ * never asks (scoreStep): it holds the hidden answers.
+ */
+async function runGpu(row: QueueRow, competition: Competition): Promise {
+  if (row.gpu) {
+    const perDevice = row.gpuPerDevice ?? runnerGpuPerDevice()
+    const sliceMb = await gpuSliceFor(row.gpu, { perDevice })
+    noteQueueGpuShare(row, sliceMb)
+    return { device: row.gpu, sliceMb, perDevice }
+  }
+  if (!runnerCardsHere() && row.kind === 'notebook' && competition.limits.gpu === true) return { ...CLUSTER_GPU }
+  return undefined
+}
+
+/**
+ * The card of a run about to start that the owner has taken back from
+ * submissions since the queue took it; `null` — none: the card is still
+ * given, the run has no card, or it has the cluster's.
+ *
+ * Asked once more right before `docker run`, after the slow part: the card's
+ * size (runGpu) takes seconds on a card nobody has asked about, and before it
+ * the run's inputs were composed and its package set hashed. An owner who
+ * gave the card back to classes in between gave it to a class that needs it
+ * now. The rooms keep off a card a running row holds (kernel/pool.ts ·
+ * takeGpu), so a run started on it anyway kept the card from that class for
+ * the run's whole time limit, for a run that had not begun. Instead the run
+ * goes back to the queue for a card still given (CardWithdrawn), and the
+ * card is the classes' at once.
+ */
+function takenBack(gpu: RunRequest['gpu']): string | null {
+  return gpu?.device && !runnerGpuDevices().includes(gpu.device) ? gpu.device : null
+}
+
+/**
+ * A GPU notebook whose card the owner took back from submissions between the
+ * queue's take and `docker run` (takenBack). A resource deferral, like a Pod
+ * the cluster could not place (runJob · deferResourceAttempt): no run began,
+ * the card goes with the attempt, and the row waits for a card still given,
+ * counting toward nothing. The participant reads the queue's usual words
+ * about resources; the teacher reads ours, which name the card — not the
+ * cluster's "Pod unschedulable", which on this machine would be untrue.
+ */
+class CardWithdrawn extends CompetitionResourcePending {
+  constructor(readonly device: string) {
+    super('gpu', 'notebook')
+  }
+
+  override get teacherReason(): string {
+    return tr('competitions.answer.gpuWithdrawnTeacher', { device: this.device })
+  }
+}
+
+/**
+ * How close, either way, the watchdog's kill of a neighbour and a run's
+ * out-of-memory on the same card must be for the second to be the first's
+ * doing. The watchdog stops a run only on its second sample over the line,
+ * three seconds apart and up to five each (docker-runner.ts · VramWatch), and
+ * both runs then take a few seconds to end and be collected.
+ */
+export const NEIGHBOUR_WINDOW_MS = 20_000
+
+/**
+ * The same after the stop of a neighbour whose card could not be read
+ * (`gpu-unread`, docker-runner.ts · VRAM_UNREAD_LIMIT). Such a run held
+ * memory nobody saw for as long as its unanswered samples took — up to about
+ * twenty-five seconds, each given up at its five-second deadline on the
+ * three-second cycle, against the three to six of two samples over the line.
+ * Its burst may come at the start of that stretch and its stop at the end, so
+ * the stop may land that much later after the run it crowded out: fifteen
+ * seconds more. It is also how long a run that ran out of memory beside a
+ * neighbour still on the card waits to learn whether that neighbour is
+ * stopped (neighbourVerdict) — fifteen seconds longer than before for an
+ * out-of-memory that turns out to be its own, which nothing on this side can
+ * tell from a neighbour's in advance. That is the price of not billing the
+ * participant for a card a run nobody could read had taken.
+ */
+export const UNREAD_NEIGHBOUR_WINDOW_MS = NEIGHBOUR_WINDOW_MS + 15_000
+
+/** A neighbour stopped from outside: when, and how long after a run's end the stop may still be that run's cause. */
+export interface NeighbourStop {
+  at: number
+  windowMs: number
+}
+
+/**
+ * A run stopped from outside at `at`, as the runs beside it on its card are
+ * judged against it: for going past its share (`gpu`), or because its card
+ * could not be read (`gpu-unread`). Both count. A run whose memory nobody
+ * could read may have held any of the card meanwhile — the fail-closed rule
+ * exists for a notebook that keeps itself unreadable to take the card — and a
+ * neighbour that ran out of memory then was crowded out by it as surely as by
+ * a burst the watchdog saw; only later, by the length of its unread stretch.
+ */
+export function neighbourStop(at: number, by: 'gpu' | 'gpu-unread'): NeighbourStop {
+  return { at, windowMs: by === 'gpu-unread' ? UNREAD_NEIGHBOUR_WINDOW_MS : NEIGHBOUR_WINDOW_MS }
+}
+
+/** The stops of runs on each card, by device; only the recent ones are kept. */
+const shareKills = new Map()
+
+function noteShareKill(device: string, at: number, by: 'gpu' | 'gpu-unread'): void {
+  const recent = (shareKills.get(device) ?? []).filter((stop) => at - stop.at <= 3 * UNREAD_NEIGHBOUR_WINDOW_MS)
+  recent.push(neighbourStop(at, by))
+  shareKills.set(device, recent)
+}
+
+/**
+ * Pure: whether a run that died of torch's out-of-memory at `endedAt` on a
+ * shared card, taken at `startedAt`, was crowded out by a neighbour's stop
+ * (`crowded`), may still learn of one (`wait`: another run `stays` on the
+ * card, and a stop could yet land within its window), or died of its own
+ * (`own`). A stop counts when it came while this run ran — one before it was
+ * taken freed the memory before it asked for any — at most
+ * NEIGHBOUR_WINDOW_MS before its end and at most the stop's own window after.
+ */
+export function neighbourVerdict(
+  stops: readonly NeighbourStop[],
+  run: { startedAt: number; endedAt: number },
+  now: number,
+  stays: boolean,
+): 'crowded' | 'wait' | 'own' {
+  const since = Math.max(run.startedAt, run.endedAt - NEIGHBOUR_WINDOW_MS)
+  if (stops.some((stop) => stop.at >= since && stop.at <= run.endedAt + stop.windowMs)) return 'crowded'
+  return stays && now - run.endedAt < UNREAD_NEIGHBOUR_WINDOW_MS ? 'wait' : 'own'
+}
+
+/**
+ * Whether a run that died of torch's out-of-memory at `endedAt` on a shared
+ * card died of a neighbour (neighbourVerdict). The stop may land after the
+ * victim has already failed — its cudaMalloc fails during the burst, and the
+ * burst is stopped only on the second sample over the line, or after the last
+ * of VRAM_UNREAD_LIMIT unanswered ones — so while another run is still on the
+ * card this waits for it: up to the longest window, and no longer than that
+ * run stays (a neighbour that ends of its own accord was not stopped).
+ */
+async function neighbourOverran(row: QueueRow, device: string, endedAt: number): Promise {
+  const signal = aborters.get(row.attemptId!)?.signal
+  const run = { startedAt: row.startedAt ?? 0, endedAt }
+  for (;;) {
+    const stays = runningRows().some((other) => other.gpu === device && other.submissionId !== row.submissionId)
+    const verdict = neighbourVerdict(shareKills.get(device) ?? [], run, Date.now(), stays)
+    if (verdict !== 'wait') return verdict === 'crowded'
+    if (stopped || signal?.aborted) return false
+    await new Promise((resolve) => setTimeout(resolve, 100))
+  }
+}
+
 /**
  * The code cells of the sent notebook — what the harness counts as "cells"
  * (harness.ts · cells_total), read from the notebook we stored ourselves, not
@@ -956,11 +1221,14 @@ const MB = 1024 * 1024
  * number: "did not fit into 10 minutes" is more useful than "time limit".
  *
  * One more sentence of ours may follow the traceback: when the cell died on
- * a missing module, it says what only the platform knows (missingPackageHint).
- * `sources` is where the submission's packages could come from; without it
- * the traceback goes alone.
+ * a missing module, it says what only the platform knows (missingPackageHint),
+ * and when torch ran out of video memory on a card, the share the run had
+ * (gpuMemoryHint). `sources` is where the submission's packages could come
+ * from; without it the traceback goes alone. `gpu` is the card the run was
+ * given: a run stopped for its share of the card is told so in numbers
+ * (gpuKillNote).
  */
-export function notebookNote(outcome: RunOutcome, competition: Competition, sources: PackageSources | null = null): string | null {
+export function notebookNote(outcome: RunOutcome, competition: Competition, sources: PackageSources | null = null, gpu: RunRequest['gpu'] | null = null): string | null {
   const at = outcome.cell + 1
   switch (outcome.status) {
     case 'ok':
@@ -982,7 +1250,10 @@ export function notebookNote(outcome: RunOutcome, competition: Competition, sour
         })
       }
       const head = tr('competitions.answer.cellFailed', { cell: at, cells: outcome.cells })
-      return outcome.detail ? withinLimit(head, outcome.detail, missingPackageHint(outcome.detail, sources)) : head
+      if (!outcome.detail) return head
+      // One hint at most: a missing module and torch's out-of-memory are different last lines.
+      const hint = missingPackageHint(outcome.detail, sources) ?? gpuMemoryHint(errorTypeOf(outcome.detail), gpu)
+      return withinLimit(head, outcome.detail, hint)
     }
     case 'cell_timeout':
     case 'timeout':
@@ -998,7 +1269,7 @@ export function notebookNote(outcome: RunOutcome, competition: Competition, sour
         cells: outcome.cells,
       })
     case 'out-of-memory':
-      return tr('competitions.answer.outOfMemory', {
+      return gpuKillNote(outcome, gpu) ?? tr('competitions.answer.outOfMemory', {
         count: Math.max(1, Math.round(competition.limits.memoryMb / 1024)),
         cell: at,
       })
@@ -1161,14 +1432,101 @@ export function missingPackageHint(detail: string, sources: PackageSources | nul
     : tr('competitions.answer.missingPackage.attach', { name })
 }
 
-/** What the TEACHER reads: the exit code, OOM and the tail of the container log. */
-export function teacherNote(outcome: RunOutcome): string | null {
+/**
+ * "6", "7,1": a share or a peak in gigabytes to a tenth — what a person
+ * compares at a glance, in the instance's own decimal separator.
+ */
+const gigabytes = (mb: number): string => formatNumber(Math.round(mb / 102.4) / 10)
+
+/**
+ * What the PARTICIPANT reads about a run stopped for its share of a card;
+ * `null` — the run was not stopped for that (any other out-of-memory is the
+ * machine's memory, and keeps its own sentence).
+ *
+ * Our own words and our own numbers: the share is the card's memory over the
+ * runs sharing it, and the peak is what the watchdog read from outside the
+ * notebook (docker-runner.ts · VramWatch). Neither a traceback nor anything
+ * the notebook printed — there is none: it was killed from outside. `brief`
+ * leaves the peak out. Under the brief output policy how far past its share
+ * a run went is a number the notebook chose while it had the hidden test in
+ * memory — a channel, exactly like how much it printed (briefNote) — and only
+ * the share, which is ours, stays.
+ */
+export function gpuKillNote(outcome: RunOutcome, gpu: RunRequest['gpu'] | null, brief = false): string | null {
+  const slice = gpu?.sliceMb ?? null
+  if (outcome.diagnostics.killedBy !== 'gpu' || slice === null) return null
+  const peak = outcome.diagnostics.gpuPeakMb ?? null
+  const cell = outcome.cell + 1
+  if (brief || peak === null) return tr('competitions.answer.gpuOverShare', { slice: gigabytes(slice), cell })
+  return tr('competitions.answer.gpuOutOfMemory', { peak: gigabytes(peak), slice: gigabytes(slice), cell })
+}
+
+/**
+ * The sentence after torch's own OutOfMemoryError in a run that had a card;
+ * `null` — no card, or another error.
+ *
+ * Not the watchdog's kill (gpuKillNote): here torch met its share inside the
+ * participant's own cell — the allocator's cap (harness.ts · SITECUSTOMIZE) or
+ * a full card — and the traceback says "CUDA out of memory" with nothing about
+ * why a 24 GB card ran out at six. The share is ours: the card's memory over
+ * the runs sharing it, as the run was given it (RunRequest.gpu), never a
+ * number the notebook reported. So a blind participant may read it too, next
+ * to the type (briefNote). With a card to itself there is no share to name,
+ * and under the cluster, or on a card that did not say its size, nobody can
+ * name one — the advice stays.
+ */
+export function gpuMemoryHint(errorType: string | null, gpu: RunRequest['gpu'] | null): string | null {
+  if (!gpu || errorType !== 'OutOfMemoryError') return null
+  // The cluster's card (CLUSTER_GPU): its device plugin decides the share, not us.
+  if (gpu.device === '') return tr('competitions.answer.gpuOomCard')
+  if (gpu.perDevice <= 1) return tr('competitions.answer.gpuOomWhole')
+  return gpu.sliceMb === null
+    ? tr('competitions.answer.gpuOomCard')
+    : tr('competitions.answer.gpuOomShare', { slice: gigabytes(gpu.sliceMb) })
+}
+
+/**
+ * What the TEACHER reads: the exit code, OOM and the tail of the container
+ * log — and for a run with a watched share of a card, a line about the card
+ * (cardNote).
+ */
+export function teacherNote(outcome: RunOutcome, gpu: RunRequest['gpu'] | null = null): string | null {
   if (outcome.status === 'ok') return null
   const head = `${outcome.status} · exit ${outcome.diagnostics.exit ?? '—'}${
     outcome.diagnostics.oomKilled ? ' · OOMKilled' : ''
   }${outcome.diagnostics.killedBy ? ` · killed by ${outcome.diagnostics.killedBy}` : ''}`
   const detail = outcome.status === 'dependency_error' ? outcome.detail : ''
-  return [head, detail, outcome.log].filter(Boolean).join('\n')
+  return [head, detail, cardNote(outcome, gpu), outcome.log].filter(Boolean).join('\n')
+}
+
+/**
+ * The teacher's line about the card of a failed run that had a watched share:
+ * which card, the share and the peak — and for a run stopped for it, the two
+ * samples that decided the kill and the line they crossed, so a teacher who
+ * thinks the share too small has the numbers to say by how much. `null` — no
+ * card, a whole card, or a share of unknown size: nobody watched, and there
+ * is nothing to say.
+ *
+ * A watched run that no sample ever saw holding memory — none answered, or
+ * every one that did listed nothing — says the peak was not measured, not
+ * "peak 0 MB": a notebook that trained on the card for minutes read as one
+ * that never touched it.
+ */
+function cardNote(outcome: RunOutcome, gpu: RunRequest['gpu'] | null): string | null {
+  if (!gpu?.device || gpu.sliceMb === null) return null
+  if (outcome.diagnostics.killedBy === 'gpu') {
+    return tr('competitions.answer.gpuKilled', {
+      device: gpu.device,
+      samples: (outcome.diagnostics.gpuSamplesMb ?? []).slice(-2).join(', '),
+      limit: gpuKillAboveMb(gpu.sliceMb),
+      slice: gpu.sliceMb,
+    })
+  }
+  const peak = outcome.diagnostics.gpuPeakMb
+  if (peak === undefined) return null
+  return typeof peak === 'number' && peak > 0
+    ? tr('competitions.answer.gpuPeak', { device: gpu.device, peak, slice: gpu.sliceMb })
+    : tr('competitions.answer.gpuPeakUnmeasured', { device: gpu.device, slice: gpu.sliceMb })
 }
 
 /**
@@ -1204,12 +1562,17 @@ export function metricNote(outcome: ScoreOutcome): string | null {
  * line says": a notebook on the hidden test can raise a class it made up and
  * named after the rows it read, and the name would carry them out whole. The
  * list caps that channel at a handful of bits.
+ *
+ * `OutOfMemoryError` is torch's (torch.OutOfMemoryError, the CUDA allocator
+ * out of room): on a shared card it is what a run meets at its share, raised
+ * in the participant's own cell (harness.ts · SITECUSTOMIZE), and the one
+ * word a blind participant needs to look at their batch size.
  */
 const ERROR_TYPES: ReadonlySet = new Set([
   'ArithmeticError', 'AssertionError', 'AttributeError', 'EmptyDataError', 'FileNotFoundError',
   'ImportError', 'IndentationError', 'IndexError', 'KeyError', 'LinAlgError', 'LookupError',
   'MemoryError', 'MergeError', 'ModuleNotFoundError', 'NameError', 'NotFittedError',
-  'NotImplementedError', 'OSError', 'OverflowError', 'ParserError', 'PermissionError',
+  'NotImplementedError', 'OSError', 'OutOfMemoryError', 'OverflowError', 'ParserError', 'PermissionError',
   'RecursionError', 'RuntimeError', 'StopIteration', 'SyntaxError', 'TypeError',
   'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'ValueError', 'ZeroDivisionError',
 ])
@@ -1243,16 +1606,27 @@ export function errorTypeOf(detail: string): string | null {
  * size or anything else the run reported about itself, never the module a
  * traceback names: each of those is text the notebook wrote while it had the
  * hidden rows in memory. The sentences that already carry nothing but our
- * numbers (time, memory, no file) are the usual ones.
+ * numbers (time, memory, no file) are the usual ones. A run stopped for its
+ * share of a card is told the share and not how far past it went
+ * (gpuKillNote); one stopped because its card could not be read, our
+ * sentence about that (unreadCardSentence).
  */
-export function briefNote(outcome: RunOutcome, competition: Competition, errorType: string | null): string | null {
+export function briefNote(outcome: RunOutcome, competition: Competition, errorType: string | null, gpu: RunRequest['gpu'] | null = null): string | null {
   const at = outcome.cell + 1
   switch (outcome.status) {
-    case 'cell_error':
+    case 'cell_error': {
       if (outcome.diagnostics.killedBy === 'output') return tr('competitions.answer.sealed.tooMuchOutput', { cell: at })
-      return errorType
+      const head = errorType
         ? tr('competitions.answer.sealed.cellFailed', { cell: at, cells: outcome.cells, type: errorType })
         : tr('competitions.answer.sealed.cellFailedUntyped', { cell: at, cells: outcome.cells })
+      // A paragraph of its own: the blind row shows only what follows the
+      // head it already repeats (web/src/lib/competition-words.ts · blindAdvice).
+      const unread = outcome.diagnostics.killedBy === 'gpu-unread' ? unreadCardSentence(outcome.detail) : null
+      const hint = unread ?? gpuMemoryHint(errorType, gpu)
+      return hint ? `${head}\n\n${hint}` : head
+    }
+    case 'out-of-memory':
+      return gpuKillNote(outcome, gpu, true) ?? notebookNote({ ...outcome, detail: '' }, competition)
     case 'target_unreadable':
       return tr('competitions.answer.unreadable', { file: SUBMISSION_FILE })
     case 'dependency_error':
@@ -1264,6 +1638,28 @@ export function briefNote(outcome: RunOutcome, competition: Competition, errorTy
   }
 }
 
+/**
+ * The docker runner's sentence for a run stopped because its card could not
+ * be read, when the detail is exactly that sentence (docker-runner.ts ·
+ * unreadCardNotes puts it where a traceback would be); `null` otherwise.
+ *
+ * Without it a blind participant read "the notebook failed at cell 5, the
+ * output is hidden" about a run no code of theirs failed in: we stopped it,
+ * and why — the card went unanswered sample after sample, most often a
+ * notebook starving its own processor — is the one thing they can act on.
+ * Matched word for word against our own catalogue rather than passed on for
+ * the kill's name: the detail is the traceback's place, and under the brief
+ * policy nothing the notebook wrote may leave. The count in it is the
+ * runner's: the limit, and a sample still in flight at the stop.
+ */
+function unreadCardSentence(detail: string): string | null {
+  for (let count = VRAM_UNREAD_LIMIT; count <= 2 * VRAM_UNREAD_LIMIT; count++) {
+    const ours = tr('competitions.answer.gpuUnread', { count })
+    if (detail === ours) return ours
+  }
+  return null
+}
+
 /** Codes of our own alignment checks whose blind words exist (competitions.answer.sealed.*). */
 const SEALED_METRIC_CODES: ReadonlySet = new Set([
   'noIdColumn', 'indexColumn', 'duplicateId', 'missingRows', 'idForm', 'emptyPredictions',
@@ -1372,6 +1768,37 @@ export async function settleSubmission(submissionId: string, timeoutMs: number):
   return queueRow(submissionId)?.state !== 'running'
 }
 
+/**
+ * Take every run of a competition off the executor before the competition
+ * is deleted: the waiting ones leave the queue, the running ones are killed
+ * the way "Kill" kills them and waited for, each for at most `settleMs`.
+ *
+ * Without it the delete took the running rows away under live containers:
+ * the run went on to its time limit holding its share of a card, while the
+ * queue, which knows a card is busy only from its rows, put the next GPU
+ * notebook on it — "the whole card" promised on a card a stranger filled.
+ * `false` — a run is still going (it did not stop in time, or it belongs to a
+ * previous life of the server that nobody here can wait for), and nothing
+ * may be deleted yet. The waiting ones stay cancelled either way, as when a
+ * participant is removed (erase.ts): the competition is going.
+ */
+export async function stopCompetitionRuns(competitionId: string, settleMs: number): Promise {
+  const rows = queueRows().filter((row) => row.competitionId === competitionId)
+  if (rows.some((row) => row.state === 'running' && !runningHere(row.submissionId))) return false
+  // Before the first await: the pump must not start them while the running ones are taken down.
+  for (const row of rows) {
+    if (row.state !== 'waiting') continue
+    leaveQueue(row.submissionId)
+    updateSubmission(row.submissionId, { state: 'cancelled', stage: 'queue', teacherError: tr('competitions.answer.killedByTeacher') })
+  }
+  for (const row of rows) {
+    if (row.state !== 'running') continue
+    await cancelSubmission(row.submissionId, 'teacher')
+    if (!(await settleSubmission(row.submissionId, settleMs))) return false
+  }
+  return true
+}
+
 /**
  * Execute the notebook again — the same as submitting it once more, but
  * without a number.
diff --git a/server/src/competitions/settings.ts b/server/src/competitions/settings.ts
index 73f21a77..203e855b 100644
--- a/server/src/competitions/settings.ts
+++ b/server/src/competitions/settings.ts
@@ -11,7 +11,11 @@
  *
  * WHAT IS NOT HERE: what `auto` means on this machine. That is a question for
  * the machine (competitions/capacity.ts), and this module stays importable
- * from store.ts without dragging the kernel modules along.
+ * from store.ts without dragging the kernel modules along. Nor what the cards
+ * given to submissions mean on this machine (competitions/gpu.ts): only which
+ * ones the owner chose — so that kernel/pool.ts, which loads before anything
+ * competition-shaped, can leave them out of the rooms' pool without an import
+ * cycle.
  */
 /*
  * The box is created by `admin/settings`, and the import is not decoration:
@@ -20,8 +24,13 @@
  */
 import '../admin/settings.js'
 import { db } from '../db.js'
-import type { CompetitionLimits } from '@shared/competitions'
-import { SETTINGS_LIMITS, type CompetitionSettingsInput } from '@shared/competitions-settings'
+import {
+  SETTINGS_LIMITS,
+  type CompetitionDefaults,
+  type CompetitionGpuSettingsInput,
+  type CompetitionSettingsInput,
+  type SettingSource,
+} from '@shared/competitions-settings'
 
 const KEY = {
   slots: 'competitions.slots',
@@ -30,6 +39,9 @@ const KEY = {
   memoryMb: 'competitions.defaults.memoryMb',
   cpus: 'competitions.defaults.cpus',
   perDay: 'competitions.defaults.perDay',
+  /** A JSON array of `KERNEL_GPUS` entries; `[]` is "none", unlike a missing row. */
+  gpuDevices: 'competitions.gpu.devices',
+  gpuPerDevice: 'competitions.gpu.perDevice',
 } as const
 
 const DEFAULT_FIELDS = ['wallSeconds', 'memoryMb', 'cpus', 'perDay'] as const
@@ -99,11 +111,11 @@ export function uploadsPerMinute(): { value: number; source: 'saved' | 'default'
  * never touched.
  */
 export function competitionDefaults(): {
-  limits: CompetitionLimits
-  sources: Record
+  limits: CompetitionDefaults
+  sources: Record
 } {
-  const limits = {} as CompetitionLimits
-  const sources = {} as Record
+  const limits = {} as CompetitionDefaults
+  const sources = {} as Record
   for (const field of DEFAULT_FIELDS) {
     const saved = storedWhole(KEY[field], SETTINGS_LIMITS[field])
     limits[field] = saved ?? SETTINGS_LIMITS[field].default
@@ -112,14 +124,101 @@ export function competitionDefaults(): {
   return { limits, sources }
 }
 
+/* ------------------------------------------------------- runner cards */
+
+/*
+ * The cards given to competition submissions, and how many runs share one.
+ *
+ * Only the SETTING lives here: the ids the owner chose, as saved or as the
+ * environment names them. What they mean on this machine — which of them
+ * KERNEL_GPUS still lists, who holds them now, how much memory a run gets —
+ * is competitions/gpu.ts's question; this half is what kernel/pool.ts needs to
+ * keep those cards out of the rooms' pool, and the pool cannot import gpu.ts.
+ *
+ * The same order as every setting of the box: the owner's saved choice, then
+ * the environment (COMPETITION_GPUS, COMPETITION_GPU_PER_DEVICE — how an
+ * instance is configured through .env), then the default: no card, one run
+ * per card. No card by default, and that is the point: a machine with cards
+ * keeps giving every one of them to rooms, exactly as it did before
+ * submissions could have one, until the owner moves a card over.
+ */
+
+/** A device list as KERNEL_GPUS is written: comma-separated, trimmed, each id once. */
+export function parseDeviceList(raw: string | undefined): string[] {
+  return [...new Set((raw ?? '').split(',').map((device) => device.trim()).filter((device) => device.length > 0))]
+}
+
+/**
+ * Which cards are given to submissions: the saved list, then
+ * COMPETITION_GPUS, then none. Pure, so the order is checked without a
+ * database. A saved `[]` is an answer ("none"), not a missing row: an owner
+ * who took every card back from submissions must not have .env hand them out
+ * again.
+ */
+export function resolveRunnerGpus(input: { saved: string[] | null; env?: string }): { devices: string[]; source: SettingSource } {
+  if (input.saved !== null) return { devices: input.saved, source: 'saved' }
+  const fromEnv = parseDeviceList(input.env)
+  if (fromEnv.length > 0) return { devices: fromEnv, source: 'env' }
+  return { devices: [], source: 'default' }
+}
+
+/**
+ * Runs per card: the saved number, then COMPETITION_GPU_PER_DEVICE, then one.
+ * A typo in .env is passed over rather than obeyed; a number past the bounds is
+ * pulled into them, the way COMPETITION_SLOTS is.
+ */
+export function resolveGpuPerDevice(input: { saved: number | null; env?: string }): { value: number; source: SettingSource } {
+  const range = SETTINGS_LIMITS.gpuPerDevice
+  if (input.saved !== null) return { value: input.saved, source: 'saved' }
+  const raw = (input.env ?? '').trim()
+  if (/^\d+$/.test(raw)) return { value: Math.min(range.max, Math.max(range.min, Number(raw))), source: 'env' }
+  return { value: range.default, source: 'default' }
+}
+
+/** What the owner saved; `null` — nothing (or a row nobody can read), the environment decides. */
+export function savedRunnerGpus(): string[] | null {
+  const raw = stored(KEY.gpuDevices)
+  if (raw === null) return null
+  try {
+    const parsed: unknown = JSON.parse(raw)
+    if (!Array.isArray(parsed) || parsed.some((device) => typeof device !== 'string' || !device.trim())) return null
+    return [...new Set(parsed.map((device: string) => device.trim()))]
+  } catch {
+    return null
+  }
+}
+
+/**
+ * The cards given to submissions as the setting names them — NOT yet checked
+ * against KERNEL_GPUS (competitions/gpu.ts · runnerGpuDevices does that).
+ * Read on every call, like every setting of the box: kernel/pool.ts asks it
+ * at each room start, and a stale answer would hand a submissions' card to a
+ * class.
+ */
+export function runnerGpuSetting(env: NodeJS.ProcessEnv = process.env): { devices: string[]; source: SettingSource } {
+  return resolveRunnerGpus({ saved: savedRunnerGpus(), env: env.COMPETITION_GPUS })
+}
+
+/** How many runs share one runner card, and where the number came from. */
+export function gpuPerDeviceSetting(env: NodeJS.ProcessEnv = process.env): { value: number; source: SettingSource } {
+  return resolveGpuPerDevice({
+    saved: storedWhole(KEY.gpuPerDevice, SETTINGS_LIMITS.gpuPerDevice),
+    env: env.COMPETITION_GPU_PER_DEVICE,
+  })
+}
+
 /* ------------------------------------------------------------ the form */
 
 /** A checked form: a value to store, or `null` to forget the stored one. */
-export type SettingsPatch = CompetitionSettingsInput
+export type SettingsPatch = CompetitionSettingsInput & CompetitionGpuSettingsInput
 
 export type SettingsRefusal =
   | { field: string; why: 'range'; min: number; max: number }
   | { field: string; why: 'value' }
+  /** A card that is not an entry of KERNEL_GPUS. */
+  | { field: 'gpuDevices'; why: 'device'; device: string }
+  /** Under Kubernetes the cluster sets the submissions' cards, not this form. */
+  | { field: 'gpuDevices' | 'gpuPerDevice'; why: 'managed' }
 
 /**
  * Read the owner's form against the bounds AND the machine.
@@ -128,11 +227,12 @@ export type SettingsRefusal =
  * a competition every submission of which dies of memory, and eight CPUs on a
  * docker that has four is a `docker run` refused in the middle of a class.
  * Pure: the machine comes in as numbers (capacity.ts · machineShape), so the
- * rules are checked without one.
+ * rules are checked without one. Its cards come in the same way: `gpus` is
+ * KERNEL_GPUS (none when absent), and `gpuManaged` says the cluster has them.
  */
 export function parseSettingsInput(
   body: unknown,
-  machine: { cpus: number; memoryMb: number; reserveMb: number },
+  machine: { cpus: number; memoryMb: number; reserveMb: number; gpus?: readonly string[]; gpuManaged?: boolean },
 ): { patch: SettingsPatch } | { refusal: SettingsRefusal } {
   const raw = (body ?? {}) as Record
   const has = (key: string) => Object.prototype.hasOwnProperty.call(raw, key)
@@ -176,6 +276,47 @@ export function parseSettingsInput(
     }
     patch[field] = value
   }
+
+  /*
+   * The cards. Only a card the server already gives out: an entry of
+   * KERNEL_GPUS, spelled as there. Docker gets the id as written, and the same
+   * card named "GPU-…" here while KERNEL_GPUS calls it "0" would be one card
+   * handed out twice — to a class and to submissions — with no isolation
+   * between them. The list is stored in KERNEL_GPUS order, each card once.
+   *
+   * Under Kubernetes the cluster has the cards (RUNTIME_COMPETITION_GPU_*),
+   * and a value saved here would be a promise nothing keeps; forgetting one
+   * (`null`) is still allowed, it changes nothing.
+   */
+  if (has('gpuDevices')) {
+    if (cleared(raw.gpuDevices)) patch.gpuDevices = null
+    else {
+      if (machine.gpuManaged) return { refusal: { field: 'gpuDevices', why: 'managed' } }
+      const list = raw.gpuDevices
+      if (!Array.isArray(list) || list.some((device) => typeof device !== 'string' || !device.trim())) {
+        return { refusal: { field: 'gpuDevices', why: 'value' } }
+      }
+      const known = [...new Set(machine.gpus ?? [])]
+      const chosen = new Set((list as string[]).map((device) => device.trim()))
+      for (const device of chosen) {
+        if (!known.includes(device)) return { refusal: { field: 'gpuDevices', why: 'device', device } }
+      }
+      patch.gpuDevices = known.filter((device) => chosen.has(device))
+    }
+  }
+  if (has('gpuPerDevice')) {
+    if (cleared(raw.gpuPerDevice)) patch.gpuPerDevice = null
+    else {
+      if (machine.gpuManaged) return { refusal: { field: 'gpuPerDevice', why: 'managed' } }
+      const value = whole(raw.gpuPerDevice)
+      const range = SETTINGS_LIMITS.gpuPerDevice
+      if (value === null) return { refusal: { field: 'gpuPerDevice', why: 'value' } }
+      if (value < range.min || value > range.max) {
+        return { refusal: { field: 'gpuPerDevice', why: 'range', min: range.min, max: range.max } }
+      }
+      patch.gpuPerDevice = value
+    }
+  }
   return { patch }
 }
 
@@ -195,4 +336,12 @@ export const saveCompetitionSettings = db.transaction((patch: SettingsPatch): vo
     if (value === null) dropSetting.run(KEY[field])
     else writeSetting.run(KEY[field], String(value))
   }
+  if (patch.gpuDevices !== undefined) {
+    if (patch.gpuDevices === null) dropSetting.run(KEY.gpuDevices)
+    else writeSetting.run(KEY.gpuDevices, JSON.stringify(patch.gpuDevices))
+  }
+  if (patch.gpuPerDevice !== undefined) {
+    if (patch.gpuPerDevice === null) dropSetting.run(KEY.gpuPerDevice)
+    else writeSetting.run(KEY.gpuPerDevice, String(patch.gpuPerDevice))
+  }
 })
diff --git a/server/src/competitions/store.ts b/server/src/competitions/store.ts
index 8d3ee542..0d1dcc37 100644
--- a/server/src/competitions/store.ts
+++ b/server/src/competitions/store.ts
@@ -31,6 +31,12 @@ import { db } from '../db.js'
 import { entrantKeyDigest, newEntrantKey, sealEntrantKey, unsealEntrantKey } from './key.js'
 import { removeCompetition as removeCompetitionFiles, pruneSubmissions } from './storage.js'
 import { competitionDefaults } from './settings.js'
+/*
+ * The rule by which a GPU run picks its card, called inside `takeNext`'s
+ * transaction with the counts read there. gpu.ts reaches nothing of this
+ * module, so the import closes no cycle.
+ */
+import { cardsInUse, pickRunnerGpu } from './gpu.js'
 import { putSettingRow, settingRows } from '../admin/settings.js'
 import { instanceTimeZone } from '../time-zone.js'
 import {
@@ -353,6 +359,29 @@ ensureColumn('competition_queue', 'attempt_id', 'attempt_id TEXT')
 ensureColumn('competition_queue', 'pending_kind', 'pending_kind TEXT')
 ensureColumn('competition_queue', 'resource_retries', 'resource_retries INTEGER NOT NULL DEFAULT 0')
 ensureColumn('competition_queue', 'not_before', 'not_before INTEGER NOT NULL DEFAULT 0')
+/*
+ * The runner card a running row holds (a KERNEL_GPUS entry), written by the
+ * transaction that takes the row and cleared by every way out of 'running'.
+ * In the row and not in the runner's memory for the same reason as the queue
+ * itself: which card is busy must survive a restart in the middle of a run,
+ * and the next card is chosen by counting these (takeNext). NULL for every
+ * existing row and for every run without a card — and for a GPU run on the
+ * cluster, which places the Pod itself.
+ */
+ensureColumn('competition_queue', 'gpu', 'gpu TEXT')
+/*
+ * What that run was promised on its card, written with the card and cleared
+ * with it: how many runs shared the card when it was taken (`gpu_per_device`,
+ * 1 — the whole card) and the share in MB that makes there (`gpu_slice_mb`,
+ * filled in once the card has said its size, runner.ts · runGpu). In the row
+ * because the owner may change «посылок на карту» while runs hold a card: a
+ * share read from the live setting gave the runs taken after the change a
+ * share the card no longer had room for (36 GB promised on a 24 GB card), and
+ * the console showed the old runs with the new, smaller one. The run is built
+ * from these, and the queue weighs a card's load by them (gpu.ts · cardsInUse).
+ */
+ensureColumn('competition_queue', 'gpu_per_device', 'gpu_per_device INTEGER')
+ensureColumn('competition_queue', 'gpu_slice_mb', 'gpu_slice_mb INTEGER')
 
 // Revisions are durable database state, so direct runner/queue mutations cannot
 // bypass the live feed invalidation contract. Legacy scores have unknown inputs
@@ -385,6 +414,14 @@ ensureColumn('competitions', 'board_visibility', "board_visibility TEXT NOT NULL
  */
 ensureColumn('competitions', 'late_submissions', 'late_submissions INTEGER NOT NULL DEFAULT 0')
 ensureColumn('competitions', 'output_policy', "output_policy TEXT NOT NULL DEFAULT 'brief'")
+/*
+ * «Видеокарта» (CompetitionLimits.gpu): the notebook runs with a share of a
+ * runner card. Zero for every existing row: a competition made on a GPU
+ * environment before this switch ran on the CPU, and keeps doing so until its
+ * teacher turns the card on — its submissions must not change speed by
+ * themselves in the middle of the competition.
+ */
+ensureColumn('competitions', 'gpu', 'gpu INTEGER NOT NULL DEFAULT 0')
 /*
  * A late submission and the moment its upload began (Submission.late,
  * Submission.sentAt). Every older row reads on time and without a start: late
@@ -549,6 +586,22 @@ db.exec(`
     OR NEW.memory_mb IS NOT OLD.memory_mb OR NEW.cpus IS NOT OLD.cpus
   BEGIN UPDATE competitions SET input_revision = input_revision + 1 WHERE id = NEW.id; END;
 `)
+/*
+ * The card is a limit like the CPUs: a sample notebook that passed on the CPU
+ * says nothing about the same notebook with a card (a `.cuda()` that had no
+ * device to fail on, a time limit met only with one), so turning the card on
+ * or off makes the check stale exactly as a changed CPU count does — both
+ * revisions, the notebook's and the whole input's. A trigger of its own, with
+ * a name of its own: the two above were created with IF NOT EXISTS on every
+ * existing database and cannot learn a new column.
+ */
+db.exec(`
+  CREATE TRIGGER IF NOT EXISTS competition_gpu_inputs_changed
+  AFTER UPDATE OF gpu ON competitions
+  WHEN NEW.gpu IS NOT OLD.gpu
+  BEGIN UPDATE competitions SET notebook_input_revision = notebook_input_revision + 1,
+    input_revision = input_revision + 1 WHERE id = NEW.id; END;
+`)
 /*
  * What the notebook reads is the open files AND the hidden test: a change to
  * either makes the sample notebook's check stale. The triggers carry new names
@@ -628,6 +681,7 @@ interface CompetitionRow {
   memory_mb: number
   cpus: number
   per_day: number
+  gpu: number
   environment: string
   starts_at: number | null
   deadline_at: number | null
@@ -668,6 +722,7 @@ function toCompetition(row: CompetitionRow): Competition {
       memoryMb: row.memory_mb,
       cpus: row.cpus,
       perDay: row.per_day,
+      gpu: row.gpu === 1,
     },
     environment: row.environment,
     startsAt: row.starts_at,
@@ -693,12 +748,12 @@ function toCompetition(row: CompetitionRow): Competition {
 const insertCompetition = db.prepare(`
   INSERT INTO competitions
     (id, slug, title, blurb, description, state, metric_name, metric_direction, metric_code,
-     public_percent, split_seed, wall_seconds, memory_mb, cpus, per_day, environment,
+     public_percent, split_seed, wall_seconds, memory_mb, cpus, per_day, gpu, environment,
      starts_at, deadline_at, private_release, scoring, board_visibility, late_submissions, output_policy,
      created_by, course_id, created_at, updated_at)
   VALUES
     (@id, @slug, @title, @blurb, @description, 'draft', @metric_name, @metric_direction,
-     @metric_code, @public_percent, @split_seed, @wall_seconds, @memory_mb, @cpus, @per_day,
+     @metric_code, @public_percent, @split_seed, @wall_seconds, @memory_mb, @cpus, @per_day, @gpu,
      @environment, @starts_at, @deadline_at, @private_release, @scoring, @board_visibility,
      @late_submissions, @output_policy, @created_by, @course_id, @at, @at)
 `)
@@ -760,6 +815,8 @@ export function createCompetition(input: NewCompetition): Competition | null {
       memory_mb: input.limits?.memoryMb ?? defaults.memoryMb,
       cpus: input.limits?.cpus ?? defaults.cpus,
       per_day: input.limits?.perDay ?? defaults.perDay,
+      // Never an instance default: a card is the teacher's switch, on a GPU environment.
+      gpu: input.limits?.gpu ? 1 : 0,
       environment: input.environment ?? 'base',
       starts_at: input.startsAt ?? null,
       deadline_at: input.deadlineAt ?? null,
@@ -847,6 +904,7 @@ const COLUMN_OF: Record = {
   'limits.memoryMb': 'memory_mb',
   'limits.cpus': 'cpus',
   'limits.perDay': 'per_day',
+  'limits.gpu': 'gpu',
 }
 
 const CLIP_OF: Record = {
@@ -977,9 +1035,20 @@ export function openPrivateBoard(id: string, at = Date.now()): boolean {
  * The files are removed AFTER the transaction, and deliberately out loud: a
  * directory left behind quietly means the answers of a deleted competition
  * lying on disk for an unlimited time, with nobody to notice.
+ *
+ * `'running'` — a run of it is still going, and nothing is deleted. Its
+ * queue row is what holds its slot and its share of a runner card; deleted
+ * under a live container, it let the next GPU notebook onto a card the
+ * container still filled, promised a share nobody had. The door stops the
+ * runs first (runner.ts · stopCompetitionRuns); this is the last check, in
+ * the same transaction as the delete, so a run that started in between
+ * refuses the whole of it.
  */
-export function deleteCompetition(id: string): boolean {
-  const rows = db.transaction((competitionId: string) => {
+export function deleteCompetition(id: string): boolean | 'running' {
+  const rows = db.transaction((competitionId: string): number | 'running' => {
+    if (db.prepare("SELECT 1 FROM competition_queue WHERE competition_id = ? AND state = 'running' LIMIT 1").get(competitionId)) {
+      return 'running'
+    }
     db.prepare(
       `DELETE FROM submission_runs WHERE submission_id IN
          (SELECT id FROM submissions WHERE competition_id = ?)`,
@@ -992,6 +1061,7 @@ export function deleteCompetition(id: string): boolean {
     db.prepare('DELETE FROM competition_entrants WHERE competition_id = ?').run(competitionId)
     return deleteCompetitionRow.run(competitionId).changes
   })(id)
+  if (rows === 'running') return 'running'
   if (rows === 0) return false
   removeCompetitionFiles(id)
   return true
@@ -2275,6 +2345,23 @@ export interface QueueRow {
   pendingKind: RunKind | null
   /** Its submission is late: on-time work goes first (see the selection order). */
   late: boolean
+  /**
+   * The runner card this run holds, from the moment it was taken until the
+   * attempt ends, metric step included; `null` — no card of ours: a run on
+   * the CPU, a waiting row, or a GPU run the cluster places itself.
+   */
+  gpu: string | null
+  /**
+   * How many runs shared that card when this one took it — its share is one
+   * of them (1: the whole card, uncapped). Pinned at the take, whatever the
+   * owner sets afterwards; `null` with no card.
+   */
+  gpuPerDevice: number | null
+  /**
+   * The share in MB that makes on that card; `null` with no card, until the
+   * card has said its size (runner.ts · runGpu), or when it cannot.
+   */
+  gpuSliceMb: number | null
 }
 
 interface QueueRowRaw {
@@ -2294,6 +2381,11 @@ interface QueueRowRaw {
   attempt_id: string | null
   pending_kind: string | null
   late: number
+  gpu: string | null
+  gpu_per_device?: number | null
+  gpu_slice_mb?: number | null
+  /** Only from `selectNext`: whether this work needs a card share (NEEDS_CARD). */
+  needs_gpu?: number
 }
 
 function toQueueRow(row: QueueRowRaw): QueueRow {
@@ -2314,9 +2406,16 @@ function toQueueRow(row: QueueRowRaw): QueueRow {
     attemptId: row.attempt_id,
     pendingKind: row.pending_kind as RunKind | null,
     late: row.late === 1,
+    gpu: row.gpu ?? null,
+    gpuPerDevice: row.gpu_per_device ?? null,
+    gpuSliceMb: row.gpu_slice_mb ?? null,
   }
 }
 
+/** A row letting its card go: the card and what was promised on it go together. */
+const NO_CARD = 'gpu = NULL, gpu_per_device = NULL, gpu_slice_mb = NULL'
+
+// A waiting row never holds a card: a row put back into waiting lets its card go.
 const upsertQueue = db.prepare(`
   INSERT INTO competition_queue
     (submission_id, competition_id, entrant_id, kind, state, enqueued_at, turn, late)
@@ -2334,7 +2433,8 @@ const upsertQueue = db.prepare(`
     boot = NULL,
     container = NULL,
     attempt_id = NULL,
-    pending_kind = NULL
+    pending_kind = NULL,
+    ${NO_CARD}
   WHERE competition_queue.state != 'running'
 `)
 /* Running on top, the waiting after it in arrival order — as the panel reads it. */
@@ -2351,12 +2451,13 @@ const countWaiting = db.prepare(
 const markRunning = db.prepare(`
   UPDATE competition_queue
   SET state = 'running', started_at = @at, attempts = attempts + 1, boot = @boot,
-      container = @container, attempt_id = @attempt_id
+      container = @container, attempt_id = @attempt_id, gpu = @gpu,
+      gpu_per_device = @gpu_per_device, gpu_slice_mb = NULL
   WHERE submission_id = @submission_id
 `)
 const markWaiting = db.prepare(`
   UPDATE competition_queue
-  SET state = 'waiting', started_at = NULL, boot = NULL, container = NULL, attempt_id = NULL, not_before = 0
+  SET state = 'waiting', started_at = NULL, boot = NULL, container = NULL, attempt_id = NULL, not_before = 0, ${NO_CARD}
   WHERE submission_id = ?
 `)
 const noteContainer = db.prepare(
@@ -2407,6 +2508,17 @@ export function waitingCount(): number {
   return (countWaiting.get() as { n: number }).n
 }
 
+/*
+ * Work that needs a share of a runner card: a NOTEBOOK run of a competition
+ * whose card is on (CompetitionLimits.gpu). A rescore runs only the metric,
+ * and the metric never gets a card — it holds the hidden answers — so it
+ * waits for none. Read from the competition at the take, like everything the
+ * order reads: a teacher who turns the card off lets the waiting notebooks go
+ * on the CPU at once.
+ */
+const NEEDS_CARD = `(q.kind = 'notebook' AND EXISTS
+  (SELECT 1 FROM competitions c WHERE c.id = q.competition_id AND c.gpu = 1))`
+
 /**
  * Who is next — fairly per person.
  *
@@ -2431,13 +2543,22 @@ export function waitingCount(): number {
  * is outside the standings, and the final results wait for every on-time
  * notebook and every rescore of one, not for fixes sent after the deadline.
  *
+ * And one filter, which is not a key: work that needs a share of a runner
+ * card (NEEDS_CARD) is passed over while no share is free (`@card_free`), and
+ * the next work in the order is taken instead. Passed over, not waited for:
+ * a GPU notebook waiting for a card must not hold up the CPU work behind it,
+ * and CPU work never makes a GPU notebook wait for anything but a general
+ * slot — the moment a share frees, the GPU notebook is first again in the
+ * place the keys give it. panel.ts · queueEtas simulates exactly this.
+ *
  * Computed in SQL, not in memory, because the queue is database state, and
  * two processes (the server and, one day, a separate runner) must see one and
  * the same answer.
  */
 const selectNext = db.prepare(`
-  SELECT q.* FROM competition_queue q
+  SELECT q.*, ${NEEDS_CARD} AS needs_gpu FROM competition_queue q
   WHERE q.state = 'waiting' AND q.not_before <= @at
+    AND (@card_free = 1 OR NOT ${NEEDS_CARD})
   ORDER BY
     q.late ASC,
     (SELECT COUNT(*) FROM competition_queue r
@@ -2448,11 +2569,96 @@ const selectNext = db.prepare(`
   LIMIT 1
 `)
 
-export function nextQueueRow(at=Date.now()): QueueRow | null {
-  const row = selectNext.get({at}) as QueueRowRaw | undefined
+/**
+ * The runner cards, as the queue takes work that needs one
+ * (competitions/gpu.ts · runnerGpuState): each card with how many runs it may
+ * hold now — none while a class still holds it — and the containers the
+ * runner still has to remove, card → how many (Capacity.heldGpus): such a
+ * container keeps its share until it is gone.
+ *
+ * A card's `capacity` is the runs per card a new run is taken at, and the
+ * take writes it into the row (QueueRow.gpuPerDevice): what the card already
+ * holds is weighed by the shares its runs were given, not counted by heads
+ * (gpu.ts · cardsInUse), so a card whose runs were taken at one run per card
+ * stays full after the owner raises the number, until they end.
+ *
+ * `cluster` — under the cluster: how many GPU notebooks of this instance may
+ * run at once (the broker's competition.gpuSlots, runner.ts · queueCards).
+ * The queue keeps no card there, and counts its running GPU notebooks
+ * instead: a GPU notebook taken past that number held a general slot for half
+ * a minute while its Pod waited for a unit, and came back to the head of the
+ * queue, over and over, while the CPU work behind it waited.
+ *
+ * Absent, the queue takes GPU work like any other and gives it no card: for
+ * callers from before cards.
+ */
+export interface QueueGpu {
+  cards: ReadonlyArray<{ id: string; capacity: number }>
+  held?: Readonly>
+  cluster?: number
+}
+
+const selectCardLoad = db.prepare(`
+  SELECT gpu, gpu_per_device FROM competition_queue
+  WHERE state = 'running' AND gpu IS NOT NULL
+`)
+
+const countClusterGpuRuns = db.prepare(`
+  SELECT COUNT(*) AS n FROM competition_queue q
+  WHERE q.state = 'running' AND q.gpu IS NULL AND ${NEEDS_CARD}
+`)
+
+/**
+ * Where the next GPU run would go right now: a card and the runs per card it
+ * is taken at, or — under the cluster — no card of ours (`device: null`);
+ * `null` — no share is free. Counted from the running rows at this very
+ * moment, not from the pump's picture at the start of its pass: each take
+ * writes its card into its row, so the second take of a pass already sees
+ * the first.
+ */
+function freeShare(gpu: QueueGpu): { device: string | null; perDevice: number | null } | null {
+  if (gpu.cluster !== undefined) {
+    return (countClusterGpuRuns.get() as { n: number }).n < gpu.cluster ? { device: null, perDevice: null } : null
+  }
+  const runs = (selectCardLoad.all() as Array<{ gpu: string; gpu_per_device: number | null }>)
+    .map((row) => ({ device: row.gpu, perDevice: row.gpu_per_device }))
+  const device = pickRunnerGpu(gpu.cards, cardsInUse(gpu.cards, runs, gpu.held))
+  if (device === null) return null
+  return { device, perDevice: gpu.cards.find((card) => card.id === device)?.capacity ?? null }
+}
+
+/** `@card_free` of the selection: without cards to count there is nothing to filter by. */
+function cardFree(gpu: QueueGpu | undefined, share: unknown): 0 | 1 {
+  return gpu === undefined || share !== null ? 1 : 0
+}
+
+/**
+ * The work the next take would start, without taking it — what the pump
+ * admits for memory before it takes. `gpu` must be the one the take gets:
+ * the same filter over the same rows names the same work, and an admission
+ * for one job followed by the take of another is a reservation sized for
+ * the wrong submission.
+ */
+export function nextQueueRow(at = Date.now(), gpu?: QueueGpu): QueueRow | null {
+  const row = selectNext.get({ at, card_free: cardFree(gpu, gpu ? freeShare(gpu) : null) }) as QueueRowRaw | undefined
   return row ? toQueueRow(row) : null
 }
 
+const selectCardWork = db.prepare(`
+  SELECT 1 FROM competition_queue q
+  WHERE q.state = 'waiting' AND q.not_before <= @at AND ${NEEDS_CARD}
+  LIMIT 1
+`)
+
+/**
+ * Whether any work that may be taken now needs a card share. The pump asks
+ * before it counts the cards: the count is a census of the rooms' containers
+ * (docker ps), and a queue of CPU work has no use for it.
+ */
+export function cardWorkWaiting(at = Date.now()): boolean {
+  return selectCardWork.get({ at }) !== undefined
+}
+
 /**
  * Take the next work and mark it as ours.
  *
@@ -2462,16 +2668,23 @@ export function nextQueueRow(at=Date.now()): QueueRow | null {
  * away from the class rather than speeding up the queue.
  *
  * In a transaction, because "look and take" as two queries means two copies
- * of one run on the day there are two runners.
+ * of one run on the day there are two runners. The card is chosen in it for
+ * the same reason (`gpu`, see QueueGpu): with the counts read here, by the
+ * one rule (competitions/gpu.ts · pickRunnerGpu) — two takes that each saw
+ * "one share left" on a card would otherwise both put a run on it. The card
+ * goes only to work that needs one; any other work takes none, whatever is
+ * free.
  */
 export const takeNext = db.transaction(
-  (opts: { boot?: string; at?: number; slots?: number } = {}): QueueRow | null => {
+  (opts: { boot?: string; at?: number; slots?: number; gpu?: QueueGpu } = {}): QueueRow | null => {
     if (queuePaused()) return null
     const slots = opts.slots ?? 1
     if ((selectRunning.all() as QueueRowRaw[]).length >= slots) return null
     const at = opts.at ?? Date.now()
-    const row = selectNext.get({at}) as QueueRowRaw | undefined
+    const share = opts.gpu ? freeShare(opts.gpu) : null
+    const row = selectNext.get({ at, card_free: cardFree(opts.gpu, share) }) as QueueRowRaw | undefined
     if (!row) return null
+    const card = opts.gpu && row.needs_gpu === 1 ? share : null
     markRunning.run({
       submission_id: row.submission_id,
       at,
@@ -2482,11 +2695,25 @@ export const takeNext = db.transaction(
       // made every submission on the broker fail at once as "Invalid submission
       // job identity"; the k3s smoke named its attempts itself and never saw it.
       attempt_id: randomUUID().replaceAll('-', ''),
+      gpu: card?.device ?? null,
+      // The share is pinned with the card: the run is built from it, whatever the owner sets next.
+      gpu_per_device: card?.device ? card.perDevice : null,
     })
     return toQueueRow(selectQueueRow.get(row.submission_id) as QueueRowRaw)
   },
 )
 
+/**
+ * The share in MB a running row's card makes of its pinned runs per card,
+ * once the card has said its size (runner.ts · runGpu): what the console
+ * shows next to the run, and what the run itself was capped to.
+ */
+export function noteQueueGpuShare(row: QueueRow, sliceMb: number | null): void {
+  if (!row.attemptId) return
+  db.prepare('UPDATE competition_queue SET gpu_slice_mb = ? WHERE submission_id = ? AND attempt_id = ? AND gpu IS NOT NULL')
+    .run(sliceMb, row.submissionId, row.attemptId)
+}
+
 /** Remember the container of a running run — it is what the run is killed by. */
 export function noteQueueContainer(submissionId: string, container: string | null, attemptId?: string): void {
   if (attemptId) db.prepare('UPDATE competition_queue SET container = ? WHERE submission_id = ? AND attempt_id = ?').run(container, submissionId, attemptId)
@@ -2509,7 +2736,7 @@ export const deferResourceAttempt=db.transaction((row:QueueRow,reason:string,at=
   const delay=fresh?0:Math.min(60_000,5_000*2**Math.min(4,current.resourceRetries))
   const notBefore=at+delay
   db.prepare(`UPDATE competition_queue SET kind=COALESCE(pending_kind,?),pending_kind=NULL,state='waiting',
-    started_at=NULL,boot=NULL,container=NULL,attempt_id=NULL,attempts=?,resource_retries=?,not_before=?
+    started_at=NULL,boot=NULL,container=NULL,attempt_id=NULL,${NO_CARD},attempts=?,resource_retries=?,not_before=?
     WHERE submission_id=? AND attempt_id=?`).run(nextKind,fresh?0:Math.max(0,current.attempts-1),fresh?0:current.resourceRetries+1,notBefore,row.submissionId,row.attemptId)
   // Our own words about the queue: safe to show a blind participant as they are.
   updateSubmission(row.submissionId,{state:'queued',stage:'queue',participantError:reason,briefError:reason,teacherError:teacherReason})
@@ -2520,7 +2747,8 @@ export const finishQueueAttempt = db.transaction((row: QueueRow): boolean => {
   if (!ownsQueueAttempt(row)) return false
   const current = queueRow(row.submissionId)!
   if (current.pendingKind) {
-    db.prepare("UPDATE competition_queue SET kind = pending_kind, pending_kind = NULL, state = 'waiting', started_at = NULL, container = NULL, boot = NULL, attempt_id = NULL, attempts = 0,resource_retries=0,not_before=0 WHERE submission_id = ? AND attempt_id = ?").run(row.submissionId, row.attemptId)
+    // Back to waiting for the work asked meanwhile: the card goes with the attempt that held it.
+    db.prepare(`UPDATE competition_queue SET kind = pending_kind, pending_kind = NULL, state = 'waiting', started_at = NULL, container = NULL, boot = NULL, attempt_id = NULL, ${NO_CARD}, attempts = 0,resource_retries=0,not_before=0 WHERE submission_id = ? AND attempt_id = ?`).run(row.submissionId, row.attemptId)
     updateSubmission(row.submissionId, { state: 'queued', stage: 'queue', publicScore: null, privateScore: null })
   } else db.prepare('DELETE FROM competition_queue WHERE submission_id = ? AND attempt_id = ?').run(row.submissionId, row.attemptId)
   return true
diff --git a/server/src/dependencies/service.ts b/server/src/dependencies/service.ts
index f803978e..bf5287c6 100644
--- a/server/src/dependencies/service.ts
+++ b/server/src/dependencies/service.ts
@@ -17,7 +17,7 @@ import { dependencyMessage } from './messages.js'
 import * as store from './store.js'
 import * as files from './files.js'
 import { reserveWork, workBudgetSnapshot } from '../ops/work-budget.js'
-import { competitionCapabilities, assertCompetitionCapability } from '../competitions/capabilities.js'
+import { competitionCapabilities, assertCompetitionCapability, type CompetitionNeeds } from '../competitions/capabilities.js'
 
 const events=new EventEmitter()
 events.setMaxListeners(0)
@@ -48,22 +48,39 @@ function retainedRevision(c:Competition):EnvironmentRevision|null {
  const revision=store.competitionRevision(c.id)
  return revision?.environmentName===c.environment?revision:null
 }
-export async function executionRevision(c:Competition):Promise{
+/**
+ * The competition's pinned base, pinned now if it is not yet, once the
+ * executor can run it. `needs` is what the caller is about to run: the doors
+ * that run the notebook (the upload, the baseline check) pass its card
+ * (capabilities.ts · notebookNeeds) and are refused here too when no card is
+ * given; preparing a package set and saving a draft run no notebook and ask
+ * for none. Asked with the card, a GPU competition without one refused the
+ * preparation its own panel offered (preparation.available), in the words
+ * "this competition needs a GPU".
+ */
+export async function executionRevision(c:Competition,needs:CompetitionNeeds={}):Promise{
  const retained=retainedRevision(c)
- await assertCompetitionCapability('execution',c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest)
+ await assertCompetitionCapability('execution',c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest,needs)
  if(competitionBackend()==='test')return retained
  return ensureCompetitionRevision(c)
 }
+/**
+ * Whether the base may be pinned for the panels: the executor works, or works
+ * and only the card is missing (`gpu_unavailable` comes last, on a runtime
+ * that is otherwise up) — the base is the same with a card or without, and a
+ * participant must be able to prepare packages before the owner gives one.
+ */
+const basePinnable=(execution:{available:boolean;code:string}):boolean=>execution.available||execution.code==='gpu_unavailable'
 export async function dependencyOverview(c:Competition,eid:string):Promise{
  const retained=retainedRevision(c)
- const capabilities=await competitionCapabilities(c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest)
- const revision=capabilities.execution.available?await executionRevision(c).catch(()=>retained):retained
+ const capabilities=await competitionCapabilities(c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest,{gpu:c.limits.gpu===true})
+ const revision=basePinnable(capabilities.execution)?await executionRevision(c).catch(()=>retained):retained
  return {capabilities,policy:store.policyOf(c.id),revision,draft:store.draftOf(c.id,eid),bundles:store.listBundles(c.id,eid),joined:hasJoined(c.id,eid),quota:store.preparationQuota(eid),memoryMb:c.limits.memoryMb}
 }
 export async function adminDependencyOverview(c:Competition):Promise{
  const retained=retainedRevision(c)
- const capabilities=await competitionCapabilities(c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest)
- const revision=capabilities.execution.available?await executionRevision(c).catch(()=>retained):retained
+ const capabilities=await competitionCapabilities(c.environment,competitionBackend()==='broker'?undefined:retained?.imageDigest,{gpu:c.limits.gpu===true})
+ const revision=basePinnable(capabilities.execution)?await executionRevision(c).catch(()=>retained):retained
  return {capabilities,policy:store.policyOf(c.id),revision,bundles:store.listBundles(c.id).map(b=>({...b,entrantName:getEntrant(b.entrantId)?.name??'—'}))}
 }
 export function ownBundle(c:string,eid:string,bid:string):DependencyBundle{
diff --git a/server/src/index.ts b/server/src/index.ts
index 2ca70e2b..6f106353 100644
--- a/server/src/index.ts
+++ b/server/src/index.ts
@@ -21,6 +21,7 @@ import { tr } from '@shared/i18n'
 import { startJournal, stopJournal } from './log.js'
 import { retireDatabaseKernels, stopsLocalKernelsOnExit } from './local/kernel-cleanup.js'
 import { dropLocalRoomKernel, warmRoomPerimeter } from './kernel/pool.js'
+import { watchGpuCards } from './kernel/resources.js'
 import { kernelRetirementInProgress } from './kernel/retirement.js'
 import { kernelBackend } from './kernel/runtime-client.js'
 import http from 'node:http'
@@ -471,6 +472,14 @@ server.listen(config.port, ...(bindAddr ? ([bindAddr] as const) : ([] as const))
    * does not wait an extra second (kernel/perimeter.ts).
    */
   void warmRoomPerimeter()
+  /*
+   * The cards behind KERNEL_GPUS are asked about now, in the background, and
+   * kept warm from then on (kernel/resources.ts · watchGpuCards): the owner's
+   * first look at the Resources tab after a restart, and the first GPU
+   * submission the queue picks up again below, find the card sizes known
+   * instead of waiting for a probe container.
+   */
+  watchGpuCards()
   /*
    * The competition queue also starts right away, and in this order.
    *
diff --git a/server/src/kernel/pool.ts b/server/src/kernel/pool.ts
index 56c22948..f116fbd0 100644
--- a/server/src/kernel/pool.ts
+++ b/server/src/kernel/pool.ts
@@ -1,10 +1,20 @@
 import { tr } from '@shared/i18n'
 import { kernelBackend, requireKernelIsolation, kernelRuntimeClient, runtimeEnvironment, imageRevision, RuntimeRequestError } from './runtime-client.js'
 import type { RuntimeResizeRequest, RuntimeRoom } from '@shared/runtime'
-import { sessionCpus, sessionEnvironment, sessionKernelRevision, pinSessionKernelRevision, repinSessionKernelRevision, sessionMemoryMb, sessionRowExists, storedRules } from '../db.js'
+import { db, sessionCpus, sessionEnvironment, sessionKernelRevision, pinSessionKernelRevision, repinSessionKernelRevision, sessionMemoryMb, sessionRowExists, storedRules } from '../db.js'
 import { blockKernelStarts, kernelRetirementInProgress, kernelStartsPaused } from './retirement.js'
 import { hasWorkAllocation, observedWorkMemory, releaseWorkAllocation, reserveWork, type WorkLease } from '../ops/work-budget.js'
 import { parseMemMb, perEnvironmentMemoryMb, resourceValue } from '../admin/resource-settings.js'
+/*
+ * The cards the owner gave to competition submissions, as the setting names
+ * them, and whether submissions run on this machine's cards at all. Both
+ * modules drag in nothing but the settings box and types (they are written to
+ * stay importable from here); everything else about runner cards lives in
+ * competitions/gpu.ts, which imports this pool and so must never be imported
+ * back. Read only inside functions, at the moment a room asks for a card.
+ */
+import { runnerGpuSetting } from '../competitions/settings.js'
+import { competitionBackend } from '../competitions/runner-port.js'
 /** Production starts fixed, isolated Pods through the private runtime broker.
  * The direct Docker adapter is retained only for explicit local development.
  * Neither backend falls back to a shared Jupyter server. */
@@ -766,6 +776,43 @@ export function gpuDevices(): string[] {
     .filter((device) => device.length > 0)
 }
 
+/**
+ * The cards rooms may take: KERNEL_GPUS without the ones the owner gave to
+ * competition submissions (the Resources tab, or COMPETITION_GPUS).
+ *
+ * A card goes either to classes or to submissions, never to both: docker does
+ * not refuse a second container on a card, and a submission's runaway
+ * training sharing the lecture's card takes the lecture's video memory with
+ * it. So a card given to submissions leaves the rooms' pool at once — for NEW
+ * holders. A class that already holds it keeps it until its container ends
+ * (`takeGpu`), and submissions wait for it until then (competitions/gpu.ts ·
+ * runnerGpuState): taking a card out from under a lesson would recreate its
+ * container and wipe its variables.
+ *
+ * When submissions run on the cluster, not here, no card of this machine is
+ * theirs (competitions/gpu.ts · runnerGpuDevices says the same), and none is
+ * withheld from the rooms for nothing.
+ */
+export function roomGpuDevices(): string[] {
+  if (!submissionsUseLocalCards()) return gpuDevices()
+  const runners = new Set(runnerGpuSetting().devices)
+  return gpuDevices().filter((device) => !runners.has(device))
+}
+
+/**
+ * Whether competition submissions run on this machine's cards: the docker
+ * runner and its test stand-in do; under the broker the cluster has the
+ * cards. A backend that does not resolve (competitionBackend throws on a
+ * misconfiguration) uses none — the queue says so louder elsewhere.
+ */
+export function submissionsUseLocalCards(): boolean {
+  try {
+    return competitionBackend() !== 'broker'
+  } catch {
+    return false
+  }
+}
+
 /**
  * Slices handed out in this process that have not yet become a label on a
  * container.
@@ -788,6 +835,11 @@ const reserved = new Map()
  * (the operator rewrote the list), however, does not count as the room's own:
  * the device may no longer be on the machine.
  *
+ * `keep` widens only "the same one": the cards a room may go on holding
+ * (all of KERNEL_GPUS), while a new card comes only from `devices` (the
+ * rooms' pool). A card the owner has since given to competition submissions
+ * stays with the class that holds it, and is never handed to a new one.
+ *
  * A pure function, because this rule is exactly what has to be proven, and
  * there is no real docker in the tests.
  */
@@ -795,8 +847,9 @@ export function pickGpu(
   devices: string[],
   taken: Array>,
   sessionId: string,
+  keep: string[] = devices,
 ): string | null {
-  const mine = taken.find((held) => held.session === sessionId && devices.includes(held.gpu))
+  const mine = taken.find((held) => held.session === sessionId && keep.includes(held.gpu))
   if (mine) return mine.gpu
   const busy = new Set(taken.map((held) => held.gpu).filter((gpu) => gpu.length > 0))
   return devices.find((device) => !busy.has(device)) ?? null
@@ -811,11 +864,25 @@ export function pickGpu(
  * environment are built for CUDA, `import torch` goes through, and the first
  * `.cuda()` in the middle of the lesson says something about the driver, and
  * nobody connects that with the slice being taken.
+ *
+ * `devices` is the rooms' pool, and `toSubmissions` how many cards of
+ * KERNEL_GPUS the owner gave to competition submissions: "none allocated" on
+ * a machine whose every card went to submissions would send the person to
+ * edit .env, while the cards are one click away in the Resources tab.
+ *
+ * `finishing` is how many cards of the pool a competition submission still
+ * holds: the owner gave them back to classes while a run was on them, and
+ * the run keeps its card until it ends (`submissionGpuCards`). "All in use by
+ * other classes" sent the teacher looking for a class that was not there,
+ * while the card was a submission's for minutes more and then the classes'
+ * by itself.
  */
-export function gpuRefusal(env: string, devices: string[]): string {
+export function gpuRefusal(env: string, devices: string[], toSubmissions = 0, finishing = 0): string {
   if (devices.length === 0) {
+    if (toSubmissions > 0) return tr('server.gpu.allToSubmissions', { p0: env, p1: toSubmissions })
     return tr("server.environmentRequiresAGpuButNoneIs.53af19", { p0: env })
   }
+  if (finishing > 0) return tr('server.gpu.submissionFinishing', { p0: env, count: finishing })
   return tr("server.environmentRequiresAGpuButAllAvailable.f3d4d9", { p0: env, p1: devices.length })
 }
 
@@ -823,9 +890,14 @@ export function gpuRefusal(env: string, devices: string[]): string {
  * Who holds which slice right now, by the labels of live AND stopped
  * containers, plus this process's reservations. Needed by the panel and by
  * the allocation.
+ *
+ * `strict` throws when docker does not answer the census instead of
+ * answering "nobody": competition submissions ask with it
+ * (competitions/gpu.ts · runnerGpuState), because for them an unread census
+ * must mean "wait", not "every card is free".
  */
-export async function gpuAssignments(): Promise>> {
-  const held: Array> = (await roomContainers())
+export async function gpuAssignments(options: { strict?: boolean } = {}): Promise>> {
+  const held: Array> = (await roomContainers({ strict: options.strict }))
     /*
      * The personal-notebook container takes no part in slice allocation AT
      * ALL.
@@ -849,11 +921,116 @@ export async function gpuBusy(): Promise {
   return [...new Set((await gpuAssignments()).map((room) => room.gpu))]
 }
 
-/** A slice for the room, or a refusal. The reservation is set at once; see `reserved`. */
+/**
+ * The cards competition submissions hold right now: the ones the queue has
+ * bound to a running run (`queueBoundCards`), whether or not its container is
+ * up yet, and the ones a submission's container carries by the same
+ * `colloq.gpu` label rooms carry — live or stopped, and whichever instance on
+ * this daemon started them: a card in use is in use.
+ *
+ * Runner cards are out of the rooms' pool, so a room never meets a
+ * submission on a card — except in one window: the owner gives a card back
+ * to classes while a submission still holds it, for up to its time limit.
+ * The room census does not see a submission (it counts
+ * `colloq.kind=room-kernel` only), and `docker run` would put a lecture on the
+ * same card without a word. So a new room is kept off every card a
+ * submission holds, and the panels (routes/admin-environments.ts ·
+ * gpuState, routes/admin-resources.ts · gpuShare) promise no class such a
+ * card. Not strict, like the room census beside it: a daemon that does not
+ * answer here fails the room's own start a moment later anyway, and the
+ * queue's bindings are read whatever docker says.
+ */
+export async function submissionGpuCards(): Promise {
+  const containers = await submissionContainerCards()
+  return [...new Set([...containers, ...queueBoundCards()])]
+}
+
+/**
+ * The cards the competition queue has bound to its running runs
+ * (competition_queue.gpu), read from the database at the moment of asking.
+ *
+ * The queue writes a run's card into its row in the very transaction that
+ * takes the run (competitions/store.ts · takeNext), while the run's container
+ * carries the `colloq.gpu` label only once `docker run` is issued: seconds
+ * later, tens of seconds on the first GPU run after a restart (the run's
+ * inputs are composed, its package set hashed, its card's size probed). In
+ * between, the label census sees nothing, and a card the owner gave back to
+ * classes in that window went to a new lecture while the submission was
+ * about to start on it, uncapped at one run per card (review, 13 Oct 2026).
+ * The row is the binding from the take on; the label is still read, for a
+ * container that outlives its row.
+ *
+ * Through the database module, with a statement prepared on first use, not
+ * through the queue's store: the store imports competitions/gpu.ts, which
+ * imports this pool, and this pool loads with the server's first imports.
+ * The table is the store's to create; before it exists (a process that never
+ * loaded the store, a test of the pool alone) no row can hold a card.
+ */
+let selectBoundCards: { all(): unknown[] } | null = null
+
+function queueBoundCards(): string[] {
+  try {
+    selectBoundCards ??= db.prepare(
+      "SELECT DISTINCT gpu FROM competition_queue WHERE state = 'running' AND gpu IS NOT NULL AND gpu != ''",
+    )
+  } catch (err) {
+    if (/no such (table|column)/i.test(err instanceof Error ? err.message : String(err))) return []
+    throw err
+  }
+  return (selectBoundCards.all() as Array<{ gpu: string }>).map((row) => row.gpu)
+}
+
+/** The `colloq.gpu` labels of submissions' containers; none when docker does not answer. */
+async function submissionContainerCards(): Promise {
+  if (!(await canIsolate())) return []
+  const res = await run([
+    'ps',
+    '-a',
+    '--filter',
+    'label=colloq.kind=competition-run',
+    '--filter',
+    'label=colloq.gpu',
+    '--format',
+    '{{.Label "colloq.gpu"}}',
+  ])
+  if (res.code !== 0) return []
+  // A device id has no blanks in it; a line that does is not a label value.
+  const cards = res.out.split('\n').map((line) => line.trim())
+  return [...new Set(cards.filter((gpu) => gpu.length > 0 && gpu !== '' && !/\s/.test(gpu)))]
+}
+
+/**
+ * A slice for the room, or a refusal. The reservation is set at once; see
+ * `reserved`. A new card comes from the rooms' pool only, and never one a
+ * submission holds (`submissionGpuCards`); the card the room already holds is
+ * kept even if it has since been given to competition submissions
+ * (`roomGpuDevices`).
+ *
+ * The two docker censuses come first, and everything else is read after
+ * them, in one synchronous step with the pick and the reservation: the rooms'
+ * pool (the owner's setting), the queue's bindings, and the reservations of
+ * other rooms (the room census reads them at its very end, so it is the last
+ * thing awaited). The pool used to be read before the censuses, a docker round
+ * trip earlier: a card the owner gave to submissions in that trip, which the
+ * pump bound to a GPU run at once, still went to this room, and a card given
+ * back to classes was refused to it (review, 13 Oct 2026). Read in one step
+ * with the reservation, an owner's save lands either before it, and this room
+ * sees the new pool, or after it, and the pump's census, which reads the
+ * reservations at its end, sees this room's card as held.
+ */
 async function takeGpu(sessionId: string, env: string): Promise {
-  const devices = gpuDevices()
-  const gpu = pickGpu(devices, await gpuAssignments(), sessionId)
-  if (!gpu) throw new Error(gpuRefusal(env, devices))
+  const containers = await submissionContainerCards()
+  const rooms = await gpuAssignments()
+  const all = gpuDevices()
+  const devices = roomGpuDevices()
+  const bySubmissions = new Set([...containers, ...queueBoundCards()])
+  // A holder with no session: it can never be "the same one" for any room.
+  const submissions = [...bySubmissions].map((gpu) => ({ session: '', gpu }))
+  const gpu = pickGpu(devices, [...rooms, ...submissions], sessionId, all)
+  if (!gpu) {
+    const finishing = [...new Set(devices)].filter((device) => bySubmissions.has(device)).length
+    throw new Error(gpuRefusal(env, devices, all.length - devices.length, finishing))
+  }
   reserved.set(sessionId, gpu)
   return gpu
 }
diff --git a/server/src/kernel/resources.ts b/server/src/kernel/resources.ts
index 992e2edf..3d417cae 100644
--- a/server/src/kernel/resources.ts
+++ b/server/src/kernel/resources.ts
@@ -23,7 +23,7 @@ import os from 'node:os'
 import { spawn } from 'node:child_process'
 import { db, sessionMemoryMb } from '../db.js'
 import { listNames, needsGpu } from '../environments.js'
-import { containerLimits, defaultCpus, defaultMemoryMb, dockerRead, roomContainers, memoryLimitMb, ownLimits } from './pool.js'
+import { containerLimits, defaultCpus, defaultMemoryMb, dockerRead, gpuDevices, roomContainers, memoryLimitMb, ownLimits } from './pool.js'
 import { kernelBackend, kernelRuntimeClient } from './runtime-client.js'
 import type { GpuCard, InstanceResources, RoomResource } from '@shared/admin'
 import type { RuntimeRoom } from '@shared/runtime'
@@ -282,43 +282,51 @@ export function memoryPicture(input: {
 /* ------------------------------------------------------------------ GPU */
 
 /**
- * The machine's cards, from `nvidia-smi`, if it is there.
+ * What the local `nvidia-smi` prints for these arguments; `null` — it is not
+ * there, it failed, or it did not answer in time.
  *
- * Neither errors nor a missing binary escape: an empty list means "no cards
- * visible", and the seminar form then says nothing about GPUs. The timeout is
- * short and real (kill, not just reject): `nvidia-smi` on a machine with a
- * hung driver never returns, and this read hangs on the panel's request.
+ * Neither errors nor a missing binary escape. The timeout is short and real
+ * (kill, not just reject): `nvidia-smi` on a machine with a hung driver never
+ * returns, and this read hangs on the panel's request.
  */
-function readGpus(): Promise {
+function nvidiaSmi(args: string[]): Promise {
   return new Promise((resolve) => {
     let child: ReturnType
     try {
-      child = spawn('nvidia-smi', [
-        '--query-gpu=index,name,memory.total',
-        '--format=csv,noheader,nounits',
-      ])
+      child = spawn('nvidia-smi', args)
     } catch {
-      resolve([])
+      resolve(null)
       return
     }
     let out = ''
     let done = false
-    const finish = (cards: GpuCard[]): void => {
+    const finish = (value: string | null): void => {
       if (done) return
       done = true
       clearTimeout(timer)
-      resolve(cards)
+      resolve(value)
     }
     const timer = setTimeout(() => {
       child.kill('SIGKILL')
-      finish([])
+      finish(null)
     }, 3_000)
     child.stdout?.on('data', (chunk: Buffer) => (out += chunk.toString()))
-    child.on('error', () => finish([]))
-    child.on('close', (code) => finish(code === 0 ? parseGpus(out) : []))
+    child.on('error', () => finish(null))
+    child.on('close', (code) => finish(code === 0 ? out : null))
   })
 }
 
+/**
+ * The machine's cards, from `nvidia-smi`, if it is there.
+ *
+ * An empty list means "no cards visible", and the seminar form then says
+ * nothing about GPUs.
+ */
+async function readGpus(): Promise {
+  const out = await nvidiaSmi(['--query-gpu=index,name,memory.total', '--format=csv,noheader,nounits'])
+  return out === null ? [] : parseGpus(out)
+}
+
 /** "0, NVIDIA GeForce RTX 3090, 24576": one line per card. */
 export function parseGpus(out: string): GpuCard[] {
   const cards: GpuCard[] = []
@@ -333,6 +341,347 @@ export function parseGpus(out: string): GpuCard[] {
   return cards
 }
 
+/* ------------------------------------------- the cards behind KERNEL_GPUS */
+
+/*
+ * What each entry of KERNEL_GPUS is: the card's name and its video memory.
+ *
+ * `readGpus` above answers "what cards does this machine have" for the class
+ * form, and inside the colloq-vast container it answers nothing: nvidia-smi is
+ * not in the app's image (deploy/vast/README.md lists it as a known gap).
+ * Runner cards need more than a picture. A card's memory decides the share
+ * every submission on it is promised AND held to (competitions/gpu.ts ·
+ * gpuSliceMb), so it has to be read wherever the app actually runs. Two
+ * sources, in this order:
+ *
+ *  - the local `nvidia-smi`, when the server runs on the host: one call for
+ *    every card, matched by index or by UUID, the two ways KERNEL_GPUS names a
+ *    whole card;
+ *  - otherwise the cards themselves, asked from ONE throwaway container that
+ *    is given all of them (`docker run --gpus all --entrypoint nvidia-smi`, on
+ *    a kernel image already on the machine) and matched the same way. The
+ *    NVIDIA toolkit mounts nvidia-smi into every container it gives a card
+ *    to, and with every card visible the container numbers them as the host
+ *    does, which is how `--gpus device=` names them too. One container
+ *    for all of them, not one per card: on the 2 × 4090 VM a probe container
+ *    took ~6 s for both cards and ~7 s for each card alone (13 Oct 2026), and
+ *    a refresh in the middle of a class is then one container on a busy
+ *    daemon rather than one per card. An entry that container did not name
+ *    (it failed, or the entry is spelled in a way only `--gpus device=`
+ *    understands) is asked from a container of its own,
+ *    `--gpus device=`, whose answer is about the card docker hands out
+ *    under that very id.
+ *
+ * A MIG instance stays unknown on purpose: the per-card query describes the
+ * physical card the instance is cut from, and a share computed from that would
+ * promise each run more than the whole instance has.
+ *
+ * Read when the server starts and kept warm from then on (`watchGpuCards`),
+ * so that neither the owner's Resources tab nor the first GPU run after a
+ * restart waits for docker. Cached for ten minutes per card (one minute for
+ * "unknown", so a kernel image built a minute ago is noticed soon), one probe
+ * at a time for everyone who asks, and nothing ever throws: a card that cannot
+ * be asked is a card of unknown size, and the panel says exactly that. A card
+ * that has once said its size keeps it through a re-probe that brings none
+ * (`remembered`).
+ */
+
+/** A card's own word about itself; `null` — it could not be asked, nobody guesses. */
+export interface GpuCardFacts {
+  name: string | null
+  memoryMb: number | null
+}
+
+/** An entry of KERNEL_GPUS with what is known about it. */
+export interface KnownGpuCard extends GpuCardFacts {
+  id: string
+}
+
+/** Asks about entries of KERNEL_GPUS; an entry missing from the answer is unknown. */
+export type GpuCardProbe = (devices: readonly string[]) => Promise>
+
+const CARD_TTL_MS = 10 * 60_000
+const UNKNOWN_CARD_TTL_MS = 60_000
+const UNKNOWN_CARD: GpuCardFacts = { name: null, memoryMb: null }
+/** The test stand's card: the same made-up 3090 `collect` shows. */
+const FAKE_CARD: GpuCardFacts = { name: 'NVIDIA GeForce RTX 3090', memoryMb: 24_576 }
+/** Kernel images are `colloq-kernel:` (kernel/pool.ts · IMAGE_PREFIX). */
+const KERNEL_IMAGE = 'colloq-kernel'
+/** What the local nvidia-smi and the all-cards container are asked: one line per card (`parseCardList`). */
+const CARD_QUERY = ['--query-gpu=index,uuid,name,memory.total', '--format=csv,noheader,nounits']
+
+/** What the cache holds for a card: its facts, and when they are old enough to be asked again. */
+interface SeenCard {
+  facts: GpuCardFacts
+  until: number
+}
+
+const seenCards = new Map()
+let cardProbe: GpuCardProbe = probeGpuCards
+let cardsInflight: { devices: Set; done: Promise } | null = null
+let cardWatch: ReturnType | null = null
+
+/**
+ * Substitute the probe, for tests; `null` restores the real one. The cache
+ * goes with it either way: an answer of the old probe is not the new one's.
+ */
+export function useGpuCardProbe(fake: GpuCardProbe | null): void {
+  cardProbe = fake ?? probeGpuCards
+  seenCards.clear()
+  cardsInflight = null
+}
+
+/**
+ * The cards behind these KERNEL_GPUS entries, in the same order.
+ *
+ * A card asked about before is answered from memory at once, even when the
+ * answer is old: the old answer goes out and a fresh probe starts behind it
+ * (a card does not change its memory between two page loads). A card never
+ * asked about waits for the probe, at most `waitMs`: a participant's page
+ * must not wait half a minute for `docker run` on a cold server, and after
+ * the deadline the card is simply unknown on this answer and known on the next.
+ */
+export async function gpuCards(devices: readonly string[], options: { waitMs?: number } = {}): Promise {
+  const now = Date.now()
+  const unseen = devices.filter((device) => !seenCards.has(device))
+  const stale = devices.filter((device) => {
+    const seen = seenCards.get(device)
+    return seen !== undefined && now >= seen.until
+  })
+  if (unseen.length + stale.length > 0) {
+    const done = refreshCards([...unseen, ...stale])
+    if (unseen.length > 0) await within(done, options.waitMs ?? Infinity)
+  }
+  return devices.map((id) => ({ id, ...(seenCards.get(id)?.facts ?? UNKNOWN_CARD) }))
+}
+
+/** One probe for everyone: a caller whose cards are already being asked about joins it. */
+function refreshCards(devices: readonly string[]): Promise {
+  const flight = cardsInflight
+  if (flight && devices.every((device) => flight.devices.has(device))) return flight.done
+  const asked = new Set(devices)
+  const probe = cardProbe
+  const done: Promise = Promise.resolve()
+    .then(() => probe([...asked]))
+    .catch(() => new Map())
+    .then((answer) => {
+      // A test swapped the probe meanwhile: this answer belongs to the old one.
+      if (probe !== cardProbe) return
+      const at = Date.now()
+      for (const device of asked) seenCards.set(device, remembered(seenCards.get(device)?.facts, answer.get(device), at))
+    })
+    .finally(() => {
+      if (cardsInflight?.done === done) cardsInflight = null
+    })
+  cardsInflight = { devices: asked, done }
+  return done
+}
+
+/**
+ * What the cache keeps for a card after a probe: `known` is what it held,
+ * `heard` what the probe said (nothing — it failed, timed out or did not name
+ * the card).
+ *
+ * A size the card said goes in, replacing whatever was there, for ten
+ * minutes. A probe that brought no size does NOT erase a size the card said
+ * before: a card does not change its memory between two probes, while a probe
+ * container that times out on a daemon busy with a class's runs, or a
+ * `docker image ls` that fails once, is ordinary. Erasing it meant that every
+ * GPU run started in the next minute was promised no share at all — no
+ * allocator cap and no watchdog (competitions/gpu.ts · gpuSliceFor), on a card
+ * whose other runs were capped to theirs, so one greedy notebook could take
+ * the memory they had been promised (review, 13 Oct 2026). The kept size is
+ * asked again within a minute instead of ten, so the cache is confirmed as
+ * soon as the daemon answers. A card that never said its size stays unknown,
+ * and is asked again in a minute, as before.
+ */
+function remembered(known: GpuCardFacts | undefined, heard: GpuCardFacts | undefined, at: number): SeenCard {
+  if (heard && heard.memoryMb !== null) return { facts: heard, until: at + CARD_TTL_MS }
+  if (known && known.memoryMb !== null) return { facts: known, until: at + UNKNOWN_CARD_TTL_MS }
+  return { facts: heard ?? UNKNOWN_CARD, until: at + UNKNOWN_CARD_TTL_MS }
+}
+
+/**
+ * Keep the inventory warm from the server's start: every card of KERNEL_GPUS
+ * is asked about now, in the background, and looked at again every minute,
+ * which asks docker only about what has gone stale by the rules above (a size
+ * after ten minutes, an unknown one after one) and costs nothing otherwise.
+ *
+ * Before this, nothing asked until somebody needed an answer. The first open
+ * of the Resources tab after a restart waited for the probe containers (the
+ * owner's settings read waits seconds for a card never asked about, and the
+ * whole tab stays blank meanwhile — ~7 s on the 2 × 4090 VM), and the first
+ * GPU run the queue picked up again after a restart waited for its card the
+ * same way. Now the answer is there before anyone asks, and a request that
+ * comes while the first probe still runs joins it instead of starting another
+ * one.
+ *
+ * Only where this server gives the cards out (the docker backend: under the
+ * broker the cluster has them, and the test backend makes them up) and only
+ * on a machine whose KERNEL_GPUS lists any. Returns the way to stop it, for
+ * tests; the timer does not keep the process alive.
+ */
+export function watchGpuCards(): () => void {
+  if (cardWatch !== null || !dockerGivesCards()) return () => {}
+  // Each card once, as the owner's block asks about them: KERNEL_GPUS may name one twice.
+  const devices = [...new Set(gpuDevices())]
+  if (devices.length === 0) return () => {}
+  const look = (): void => void gpuCards(devices, { waitMs: 0 })
+  look()
+  const watch = setInterval(look, UNKNOWN_CARD_TTL_MS)
+  watch.unref?.()
+  cardWatch = watch
+  return () => {
+    clearInterval(watch)
+    if (cardWatch === watch) cardWatch = null
+  }
+}
+
+/** Whether this server hands the machine's cards out itself; a backend setting that does not parse is a "no", not a crash at start. */
+function dockerGivesCards(): boolean {
+  try {
+    return kernelBackend() === 'docker'
+  } catch {
+    return false
+  }
+}
+
+/** The work, or the deadline, whichever comes first; never rejects (the work does not). */
+function within(work: Promise, ms: number): Promise {
+  if (!Number.isFinite(ms)) return work
+  return new Promise((resolve) => {
+    const timer = setTimeout(resolve, Math.max(0, ms))
+    // A deadline is no reason to keep a finished process alive.
+    timer.unref?.()
+    void work.then(() => {
+      clearTimeout(timer)
+      resolve()
+    })
+  })
+}
+
+/**
+ * The real probe: the local nvidia-smi first, then one throwaway container
+ * given every card, then a container of its own for an entry neither named.
+ */
+async function probeGpuCards(devices: readonly string[]): Promise> {
+  const answer = new Map()
+  if (kernelBackend() === 'test' && !dockerInjected) {
+    for (const device of devices) answer.set(device, FAKE_CARD)
+    return answer
+  }
+  // Under the broker the cluster has the cards; nobody here gives them out.
+  if (kernelBackend() !== 'docker' && !dockerInjected) return answer
+  const name = (listed: CardRow[]): void => {
+    for (const device of devices) {
+      const card = listed.find((row) => row.index === device || row.uuid === device)
+      if (card && !answer.has(device)) answer.set(device, { name: card.name, memoryMb: card.memoryMb })
+    }
+  }
+  const unnamed = (): string[] => devices.filter((device) => !answer.has(device) && !device.startsWith('MIG-'))
+  const local = await nvidiaSmi(CARD_QUERY)
+  if (local !== null) name(parseCardList(local))
+  if (unnamed().length === 0) return answer
+  const image = await probeImage()
+  if (!image) return answer
+  const all = await cardsInContainer(image)
+  if (all) name(all)
+  await Promise.all(
+    unnamed().map(async (device) => {
+      const facts = await cardInContainer(device, image)
+      if (facts) answer.set(device, facts)
+    }),
+  )
+  return answer
+}
+
+/** One line of `CARD_QUERY`'s answer. */
+export type CardRow = { index: string; uuid: string; name: string; memoryMb: number | null }
+
+/**
+ * "0, GPU-8f…, NVIDIA GeForce RTX 4090, 24564": one line per card, as
+ * `--query-gpu=index,uuid,name,memory.total` prints it. A memory field that
+ * is not a number ("[N/A]") is unknown, not zero.
+ */
+export function parseCardList(out: string): CardRow[] {
+  const cards: CardRow[] = []
+  for (const line of out.split('\n')) {
+    const parts = line.split(',').map((part) => part.trim())
+    if (parts.length < 4 || !/^\d+$/.test(parts[0]) || !parts[1]) continue
+    const memoryMb = Number(parts[parts.length - 1])
+    cards.push({
+      index: parts[0],
+      uuid: parts[1],
+      name: parts.slice(2, -1).join(', '),
+      memoryMb: Number.isSafeInteger(memoryMb) && memoryMb > 0 ? memoryMb : null,
+    })
+  }
+  return cards
+}
+
+/**
+ * "NVIDIA GeForce RTX 4090, 24564" from the probe container. Docker's own
+ * words may come before it on the same stream (`run` merges stderr), so the
+ * first line of that shape is taken, not the first line.
+ */
+export function parseCardFacts(out: string): GpuCardFacts | null {
+  for (const line of out.split('\n')) {
+    const match = /^(.+?),\s*(\d+)\s*$/.exec(line.trim())
+    if (!match) continue
+    const memoryMb = Number(match[2])
+    if (Number.isSafeInteger(memoryMb) && memoryMb > 0) return { name: match[1].trim(), memoryMb }
+  }
+  return null
+}
+
+/**
+ * A kernel image to start the probe from, among those already on the
+ * machine: `base-gpu` first, it is surely there on a machine with cards. Never
+ * pulled (`--pull=never` below): `colloq-kernel` is not on any registry, and a
+ * pull attempt is half a minute of nothing.
+ */
+async function probeImage(): Promise {
+  const res = await askDocker(['image', 'ls', '--format', '{{.Repository}}:{{.Tag}}', KERNEL_IMAGE], 10_000)
+  if (res.code !== 0) return null
+  const tags = res.out
+    .split('\n')
+    .map((line) => line.trim())
+    .filter((line) => line.startsWith(`${KERNEL_IMAGE}:`) && !line.endsWith(':'))
+  return tags.find((tag) => tag === `${KERNEL_IMAGE}:base-gpu`) ?? tags.find((tag) => tag === `${KERNEL_IMAGE}:base`) ?? tags[0] ?? null
+}
+
+/**
+ * Ask the cards themselves what they are, from a container that lives a few
+ * seconds: no network, no capabilities, removed on exit. Read-only for the
+ * cards — the query takes no memory on them and disturbs no run already there.
+ */
+function probeContainer(gpus: string, image: string, query: readonly string[]): Promise<{ code: number; out: string }> {
+  return askDocker(
+    [
+      'run', '--rm', '--pull=never', '--network', 'none', '--cap-drop=ALL', '--security-opt=no-new-privileges',
+      '--gpus', gpus, '--entrypoint', 'nvidia-smi', image, ...query,
+    ],
+    30_000,
+  )
+}
+
+/**
+ * Every card at once, from one container given all of them: the same question
+ * and the same lines as the local nvidia-smi. `null` — docker did not start
+ * it or nvidia-smi failed in it; a partial answer is not read, since a line
+ * cut short by the timeout would read as a smaller card.
+ */
+async function cardsInContainer(image: string): Promise {
+  const res = await probeContainer('all', image, CARD_QUERY)
+  return res.code === 0 ? parseCardList(res.out) : null
+}
+
+/** One card, from a container given only it: for an entry the all-cards container did not name. */
+async function cardInContainer(device: string, image: string): Promise {
+  const res = await probeContainer(`device=${device}`, image, ['--query-gpu=name,memory.total', '--format=csv,noheader,nounits'])
+  return res.code === 0 ? parseCardFacts(res.out) : null
+}
+
 /* ------------------------------------------------------------------ rooms */
 
 interface RoomRow {
diff --git a/server/src/kernel/runtime-client.ts b/server/src/kernel/runtime-client.ts
index b09cfd4a..838c82e7 100644
--- a/server/src/kernel/runtime-client.ts
+++ b/server/src/kernel/runtime-client.ts
@@ -1,5 +1,5 @@
 import { tr } from '@shared/i18n'
-import { COMPETITION_JOB_ID, parseCompetitionJobIntent, parseCompetitionJobStatus, parseCompetitionJobCollection, type CompetitionJobCollection, type CompetitionJobIntent, type CompetitionJobStatus } from '@shared/competition-runtime'
+import { COMPETITION_GPU_SLOTS_MAX, COMPETITION_JOB_ID, parseCompetitionJobIntent, parseCompetitionJobStatus, parseCompetitionJobCollection, type CompetitionJobCollection, type CompetitionJobIntent, type CompetitionJobStatus } from '@shared/competition-runtime'
 /** The web process sends room intent; it never forwards a container specification. */
 import fs from 'node:fs'
 import path from 'node:path'
@@ -247,8 +247,28 @@ export class RuntimeClient {
         const row=raw as Record
         return typeof row.available==='boolean'&&typeof row.code==='string'&&row.code.length<=100&&(row.reason===null||typeof row.reason==='string'&&row.reason.length<=500)
       }
-      const competition=value.competition as Record|undefined
-      const validCompetition=competition&&capability(competition.execution)&&capability(competition.preparation)
+      /*
+       * The broker's competition capabilities, each checked, and only those.
+       * `gpu` (whether a notebook that asks for a card can run) is left out
+       * when the broker does not say it, as one from before GPU jobs does not,
+       * or says it in a shape that is not a capability: absent reads as "no",
+       * and a GPU competition is then refused at the door rather than queued
+       * for a broker that would refuse its jobs.
+       */
+      const own=(row:{available:boolean;code:string;reason:string|null})=>({available:row.available,code:row.code,reason:row.reason})
+      const reported=value.competition as Record|undefined
+      /*
+       * `gpuSlots`, how many GPU notebooks the operator says the cluster runs
+       * at once (RUNTIME_COMPETITION_GPU_SLOTS): a whole number in its
+       * bounds, or null for "not said". Anything else is left out, as an
+       * older broker leaves it out, and the queue then keeps one.
+       */
+      const slots=reported?.gpuSlots
+      const gpuSlots=slots===null||Number.isSafeInteger(slots)&&(slots as number)>=1&&(slots as number)<=COMPETITION_GPU_SLOTS_MAX?slots as number|null:undefined
+      const competition=reported&&capability(reported.execution)&&capability(reported.preparation)
+        ?{execution:own(reported.execution),preparation:own(reported.preparation),...(capability(reported.gpu)?{gpu:own(reported.gpu)}:{}),
+          ...(gpuSlots!==undefined?{gpuSlots}:{})}
+        :undefined
       return {
         ok:value.ok,reason:value.ok ? null : value.reason as string | null,
         ...(typeof cpus === 'number' ? {defaultCpus:cpus} : {}),
@@ -256,7 +276,7 @@ export class RuntimeClient {
         ...(typeof value.maxMemoryMb === 'number' ? {maxMemoryMb:value.maxMemoryMb} : {}),
         ...(typeof value.inPlaceResize === 'boolean' ? {inPlaceResize:value.inPlaceResize} : {}),
         ...(validRecovery ? { recovery } : {}),
-        ...(validCompetition ? {competition:competition as unknown as NonNullable} : {}),
+        ...(competition ? {competition} : {}),
       }
     } catch(error) { return {ok:false,reason:error instanceof Error ? error.message : tr("server.kernelRuntimeIsUnavailable.44455e")} }
   }
diff --git a/server/src/routes/admin-competitions.ts b/server/src/routes/admin-competitions.ts
index 80d9793e..47c9e392 100644
--- a/server/src/routes/admin-competitions.ts
+++ b/server/src/routes/admin-competitions.ts
@@ -29,7 +29,7 @@ import busboy from 'busboy'
 import { Router, type Request, type Response } from 'express'
 import { tr } from '@shared/i18n'
 import { acceptPinnedSubmission, executionRevision, publicExecution } from '../dependencies/service.js'
-import { assertCompetitionCapability, competitionCapabilities } from '../competitions/capabilities.js'
+import { assertCompetitionCapability, competitionCapabilities, notebookNeeds, type CompetitionNeeds } from '../competitions/capabilities.js'
 import { submissionUsesCurrentBase } from '../competitions/provenance.js'
 import { competitionRevision } from '../dependencies/store.js'
 import { DependencyStoreError } from '../dependencies/store.js'
@@ -37,7 +37,7 @@ import { dependencyMessage } from '../dependencies/messages.js'
 import { currentStaff, ownerOnly, requireStaff } from '../admin/auth.js'
 import { getTeacher } from '../admin/store.js'
 import { instanceDayStart } from '../time-zone.js'
-import { deleteEntrantFromCompetition } from '../competitions/erase.js'
+import { deleteEntrantFromCompetition, SETTLE_MS } from '../competitions/erase.js'
 import { recordAdminEvent } from '../admin/audit-log.js'
 import { canSeeCourse, isOwner, type StaffLike } from '../admin/access.js'
 import { getCourse } from '../publish/store.js'
@@ -84,6 +84,7 @@ import {
   updateSubmission,
   waitingCount,
   enqueue,
+  cardWorkWaiting,
 } from '../competitions/store.js'
 import {
   EXECUTED_FILE,
@@ -109,6 +110,7 @@ import {
 } from '../competitions/storage.js'
 import {
   executedToday,
+  gpuEnvironmentRefusal,
   medianOf,
   openRefusal,
   parseCompetitionInput,
@@ -117,10 +119,12 @@ import {
   type DoneEntry,
   type InputRefusal,
 } from '../competitions/panel.js'
-import { queueForecast } from '../competitions/forecast.js'
+import { queueForecast, runnerGpuPicture } from '../competitions/forecast.js'
 import { DEADLINE_GRACE_MS, privateBoardState } from '../competitions/results.js'
 import { machineShape, slotsResolution } from '../competitions/capacity.js'
 import { competitionDefaults, parseSettingsInput, saveCompetitionSettings, uploadsPerMinute, type SettingsRefusal } from '../competitions/settings.js'
+import { competitionGpuSliceMb, environmentNeedsGpu, gpuSettingsView, placesTaken, runnerCardsHere } from '../competitions/gpu.js'
+import { gpuDevices } from '../kernel/pool.js'
 import { HOST_RESERVE_MB } from '../kernel/resources.js'
 import { baseName, csvShape, csvUsageSplit, intakeNotebook, notebookCells } from '../competitions/intake.js'
 import {
@@ -130,6 +134,7 @@ import {
   queueSlots,
   rerunSubmission,
   rescoreCompetition,
+  stopCompetitionRuns,
   wakeCompetitionPump,
 } from '../competitions/runner.js'
 import type { QueueRow } from '../competitions/store.js'
@@ -364,8 +369,9 @@ function baselineView(competition: Competition): BaselineView | null {
 }
 
 async function viewOf(competition: Competition): Promise {
-  const capabilities = await competitionCapabilities(competition.environment, competitionRevision(competition.id)?.imageDigest)
+  const capabilities = await competitionCapabilities(competition.environment, competitionRevision(competition.id)?.imageDigest, notebookNeeds(competition))
   competition = getCompetition(competition.id) ?? competition
+  const gpuSliceMb = await competitionGpuSliceMb(competition)
   const open = listFiles(competition.id, 'open')
   const hidden = listFiles(competition.id, 'hidden')
   const hiddenViews = hidden.map((file) => fileView(competition.id, file))
@@ -394,6 +400,7 @@ async function viewOf(competition: Competition): Promise {
   return {
     competition,
     capabilities,
+    gpuSliceMb,
     openFiles: open.map((file) => fileView(competition.id, file)),
     hiddenFiles: hiddenViews,
     sealedFiles: sealedViews,
@@ -417,6 +424,25 @@ async function viewOf(competition: Competition): Promise {
 
 /* ----------------------------------------------------------------- queue */
 
+/**
+ * The card a running row holds, as the runner grid shows it.
+ *
+ * On this machine's cards: the one the queue took for it, with the share the
+ * run was given there (QueueRow.gpuSliceMb, null until the card said its
+ * size, or when it cannot) — the run's own, not what the number of runs per
+ * card in force would make now: after the owner changes it, the runs already
+ * on a card keep the share they were started with, and showing them the new
+ * one was a number nothing enforced. Under the cluster the queue keeps no
+ * card, and a GPU notebook runs on whatever unit the cluster gave it: an
+ * empty device, no share of ours. The card is held until the attempt ends,
+ * so a notebook's metric step still shows it.
+ */
+function runningCard(row: QueueRow, competition: Competition): RunningNow['gpu'] {
+  if (row.gpu !== null) return { device: row.gpu, sliceMb: row.gpuSliceMb }
+  if (!runnerCardsHere() && row.kind === 'notebook' && competition.limits.gpu === true) return { device: '', sliceMb: null }
+  return null
+}
+
 function runningNow(row: QueueRow, viewer: StaffLike | null): RunningNow | null {
   const submission = getSubmission(row.submissionId)
   const competition = getCompetition(row.competitionId)
@@ -430,6 +456,8 @@ function runningNow(row: QueueRow, viewer: StaffLike | null): RunningNow | null
      * teacher's screen, but a name and an address from a course they do not
      * teach are not theirs to read. The submission id stays — it keys the
      * row on screen and opens nothing: the kill door checks scope itself.
+     * The card stays too, for the same sum over the GPU slots: which card is
+     * busy is the machine's business, not anyone's.
      */
     return {
       submissionId: submission.id,
@@ -448,6 +476,7 @@ function runningNow(row: QueueRow, viewer: StaffLike | null): RunningNow | null
       baseline: false,
       late: row.late,
       hidden: true,
+      gpu: runningCard(row, competition),
     }
   }
   return {
@@ -466,6 +495,7 @@ function runningNow(row: QueueRow, viewer: StaffLike | null): RunningNow | null
     container: row.container,
     baseline: row.entrantId === baselineEntrantOf(competition),
     late: row.late,
+    gpu: runningCard(row, competition),
   }
 }
 
@@ -506,20 +536,41 @@ function todayStats(now: number): { done: number; averageMs: number | null } {
  * The instance queue as this viewer may see it: every count whole (waiting,
  * slots, run today), the running rows of competitions they do not run
  * blanked (`runningNow`).
+ *
+ * The runner cards come with it when any are given to submissions here
+ * (competitions/forecast.ts · runnerGpuPicture): how many GPU runs may go at
+ * once, how many of those places are taken — by the shares the runs were
+ * given, so a card still full of runs taken at fewer per card reads full
+ * (gpu.ts · placesTaken) — and the share each is promised. Under the cluster
+ * there are none to count.
+ *
+ * And with no card given at all, while GPU notebooks wait: zero places. They
+ * wait for a card nobody gave (the owner took the last one back, or the
+ * switch went on after they were sent), and without the block the console
+ * showed them as plain waiting, with no word of what for, while the queue
+ * passed them over at every take.
  */
 function queueSnapshot(viewer: StaffLike | null, now = Date.now()): QueueSnapshot {
   const pause = queuePause()
   const today = todayStats(now)
+  const running = runningRows()
+  const cards = runnerGpuPicture(now)
+  const onCards = running.flatMap((row) => row.gpu === null ? [] : [{ device: row.gpu, perDevice: row.gpuPerDevice }])
   return {
     paused: pause.paused,
     pausedAt: pause.at,
-    running: runningRows()
+    running: running
       .map((row) => runningNow(row, viewer))
       .filter((row): row is RunningNow => row !== null),
     waiting: waitingCount(),
     slots: queueSlots(),
     doneToday: today.done,
     averageMs: today.averageMs,
+    gpu: cards
+      ? { slots: cards.slots, busy: placesTaken(cards.cards, onCards), sliceMb: cards.sliceMb }
+      : runnerCardsHere() && cardWorkWaiting(Number.MAX_SAFE_INTEGER)
+        ? { slots: 0, busy: placesTaken([], onCards), sliceMb: null }
+        : null,
   }
 }
 
@@ -553,6 +604,8 @@ function waitingRows(competitionId: string | null, snapshot: QueueSnapshot, now:
       resourcePending: queued.notBefore > now,
       baseline: queued.entrantId === baselineEntrantOf(competition),
       late: queued.late,
+      // The queue's own rule of who needs a card share (store.ts · selectNext, forecast.ts · cardNeeds).
+      gpu: queued.kind === 'notebook' && competition.limits.gpu === true,
     })
   }
   return rows
@@ -580,11 +633,15 @@ function liveOf(competition: Competition, viewer: StaffLike | null, now = Date.n
  *
  * The machine goes along with the numbers: "8 slots" means nothing without
  * "on 16 CPUs and 64 GB", and the bounds for memory and CPUs are the
- * machine's, not the editor's.
+ * machine's, not the editor's. So do its cards: which of KERNEL_GPUS went to
+ * submissions, what each one is and how big a run's share comes out — read
+ * from the cards themselves, which may take a few seconds on a cold server
+ * (kernel/resources.ts · gpuCards), hence asynchronous.
  */
-function settingsView(): CompetitionSettings {
+async function settingsView(): Promise {
   const machine = machineShape()
   const { limits, sources } = competitionDefaults()
+  const gpu = await gpuSettingsView()
   return {
     slots: slotsResolution(),
     uploadsPerMinute: uploadsPerMinute(),
@@ -602,10 +659,18 @@ function settingsView(): CompetitionSettings {
       cpus: { min: SETTINGS_LIMITS.cpus.min, max: Math.max(1, Math.min(SETTINGS_LIMITS.cpus.max, machine.cpus)) },
       perDay: { min: SETTINGS_LIMITS.perDay.min, max: SETTINGS_LIMITS.perDay.max },
     },
+    gpu,
   }
 }
 
 function refuseSetting(res: Response, refusal: SettingsRefusal): void {
+  if (refusal.why === 'device') {
+    return fail(res, 400, 'invalid', tr('server.competitions.gpuUnknownDevice', { device: refusal.device }))
+  }
+  if (refusal.why === 'managed') {
+    // The same answer the environments door gives under the cluster: not a typo, the wrong place.
+    return fail(res, 409, 'managed_environment', tr('server.competitions.gpuManaged'))
+  }
   if (refusal.why === 'range') {
     return fail(res, 400, 'invalid', tr('competitions.refusal.range', { field: refusal.field, min: refusal.min, max: refusal.max }))
   }
@@ -717,6 +782,10 @@ function refuseUpload(res: Response, received: 'not-multipart' | 'cut-off'): voi
 const mb = (bytes: number) => Math.round(bytes / 1024 / 1024)
 
 function refuseInput(res: Response, refusal: InputRefusal): void {
+  // Before the field checks: this one names `limits.gpu` but is about the environment too.
+  if (refusal.why === 'cpuEnvironment') {
+    return fail(res, 400, 'invalid', tr('server.competitions.gpuCpuEnvironment', { environment: refusal.environment }))
+  }
   if (refusal.field === 'slug') {
     return fail(res, 400, 'invalid', tr(`competitions.refusal.slug.${refusal.why}`))
   }
@@ -763,8 +832,10 @@ export function adminCompetitionRoutes(): Router {
 
   /* ---------------------------------------------------------- settings */
 
-  router.get('/api/admin/competitions/settings', requireStaff, (_req, res) => {
-    res.json(settingsView())
+  router.get('/api/admin/competitions/settings', requireStaff, (_req, res, next) => {
+    settingsView()
+      .then((body) => res.json(body))
+      .catch(next)
   })
 
   /**
@@ -773,17 +844,26 @@ export function adminCompetitionRoutes(): Router {
    * By an owner: slots decide how much of the machine the queue takes from
    * the class running next to it, and the defaults become every new
    * competition's limits — both fall on people who are not in the request.
+   * The cards even more so: a card given to submissions is a card no class
+   * can start on from that moment.
    */
-  router.put('/api/admin/competitions/settings', ownerOnly('competitions.owner.settings'), (req, res) => {
+  router.put('/api/admin/competitions/settings', ownerOnly('competitions.owner.settings'), (req, res, next) => {
     const machine = machineShape()
-    const parsed = parseSettingsInput(req.body, { ...machine, reserveMb: HOST_RESERVE_MB })
+    const parsed = parseSettingsInput(req.body, {
+      ...machine,
+      reserveMb: HOST_RESERVE_MB,
+      gpus: gpuDevices(),
+      gpuManaged: !runnerCardsHere(),
+    })
     if ('refusal' in parsed) return refuseSetting(res, parsed.refusal)
     saveCompetitionSettings(parsed.patch)
     // The audit log (admin/audit-log.ts): which of the instance's competition settings, by whom.
     recordAdminEvent({ actor: currentStaff(req), action: 'settings.competitions_changed', target: { type: 'settings', id: 'competitions' }, detail: { changed: Object.keys(parsed.patch) }, req })
-    // More slots may mean work can start right now.
+    // More slots (or a card) may mean work can start right now.
     wakeCompetitionPump()
-    res.json(settingsView())
+    settingsView()
+      .then((body) => res.json(body))
+      .catch(next)
   })
 
   /**
@@ -933,6 +1013,8 @@ export function adminCompetitionRoutes(): Router {
   router.post('/api/admin/competitions', requireStaff, async (req, res) => {
     const parsed = parseCompetitionInput(req.body, { creating: true })
     if ('refusal' in parsed) return refuseInput(res, parsed.refusal)
+    const gpu = gpuEnvironmentRefusal(parsed.input, null, environmentNeedsGpu)
+    if (gpu) return refuseInput(res, gpu)
     const staff = currentStaff(req)
     if (!courseAllowed(staff, parsed.input.courseId, res)) return
     const created = createCompetition({
@@ -958,6 +1040,8 @@ export function adminCompetitionRoutes(): Router {
     if (!competition) return
     const parsed = parseCompetitionInput(req.body)
     if ('refusal' in parsed) return refuseInput(res, parsed.refusal)
+    const gpu = gpuEnvironmentRefusal(parsed.input, competition, environmentNeedsGpu)
+    if (gpu) return refuseInput(res, gpu)
     const staff = currentStaff(req)
     if (parsed.input.courseId !== undefined && parsed.input.courseId !== (competition.courseId ?? null)) {
       if (!courseAllowed(staff, parsed.input.courseId, res)) return
@@ -986,14 +1070,22 @@ export function adminCompetitionRoutes(): Router {
    *
    * By an owner: the submissions, scores and places of a hundred people
    * disappear for good, and there is nothing to restore them from.
+   *
+   * Its runs are taken off the executor first and waited for (runner.ts ·
+   * stopCompetitionRuns): a container left running holds its slot and its
+   * share of a card while the queue, its row gone, gives that share to the
+   * next GPU notebook. A run that does not stop in time refuses the delete,
+   * and the owner tries again in a moment.
    */
   router.delete(
     '/api/admin/competitions/:id',
     ownerOnly('competitions.owner.delete'),
-    (req, res) => {
+    async (req, res) => {
       const competition = competitionOf(req, res)
       if (!competition) return
-      deleteCompetition(competition.id)
+      if (!(await stopCompetitionRuns(competition.id, SETTLE_MS)) || deleteCompetition(competition.id) === 'running') {
+        return fail(res, 409, 'failed', tr('competitions.refusal.competitionRunning'))
+      }
       recordAdminEvent({ actor: currentStaff(req), action: 'competition.deleted', target: { type: 'competition', id: competition.id, label: competition.title }, req })
       res.status(204).end()
     },
@@ -1242,9 +1334,19 @@ export function adminCompetitionRoutes(): Router {
     res.json(await viewOf(getCompetition(competition.id)!))
   })
 
-  async function executionAvailable(competition: Competition, res: Response): Promise {
+  /**
+   * Whether the executor can do what this door asks of it, said in words when
+   * it cannot. `needs` is what the door runs: a door that runs the notebook
+   * (the baseline check, a rerun, opening the competition) asks for a card
+   * when the competition has one (`notebookNeeds`); a door that runs only the
+   * metric (the metric check, a rescore) asks for none — the metric never
+   * gets a card (store.ts · NEEDS_CARD). Asked with the card, a rescore after
+   * the owner gave the cards back to classes was refused "this competition
+   * needs a GPU" for work the queue would have run on the CPU at once.
+   */
+  async function executionAvailable(competition: Competition, res: Response, needs: CompetitionNeeds): Promise {
     try {
-      await assertCompetitionCapability('execution', competition.environment, competitionRevision(competition.id)?.imageDigest)
+      await assertCompetitionCapability('execution', competition.environment, competitionRevision(competition.id)?.imageDigest, needs)
       return true
     } catch (error) {
       fail(res, 503, 'failed', error instanceof Error ? error.message : tr('runtime.brokerUnavailable'))
@@ -1300,14 +1402,14 @@ export function adminCompetitionRoutes(): Router {
   router.post('/api/admin/competitions/:id/baseline/check', requireStaff, async (req, res) => {
     const competition = competitionOf(req, res)
     if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, notebookNeeds(competition))) return
     const notebook = readBaseline(competition.id, LIMITS.notebookBytes)
     if (!notebook) {
       return fail(res, 409, 'not_ready', tr('competitions.refusal.noBaselineFile'))
     }
     const entrantId = baselineEntrantOf(competition) ?? newBaselineEntrant()
     let revision
-    try { revision = await executionRevision(competition) } catch (error) {
+    try { revision = await executionRevision(competition, notebookNeeds(competition)) } catch (error) {
       // The teacher reads "check the base"; the operator needs the cause, which
       // was swallowed here (found on a bank-like cluster, 1 Oct 2026).
       console.warn(`[competitions] ${competition.id}: the environment image could not be pinned: ${error instanceof Error ? error.message.slice(0, 300) : String(error)}`)
@@ -1346,7 +1448,7 @@ export function adminCompetitionRoutes(): Router {
   router.post('/api/admin/competitions/:id/metric/check', requireStaff, async (req, res) => {
     const competition = competitionOf(req, res)
     if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, {})) return
     const baseline = competition.baselineSubmissionId
       ? getSubmission(competition.baselineSubmissionId)
       : null
@@ -1368,7 +1470,7 @@ export function adminCompetitionRoutes(): Router {
   router.post('/api/admin/competitions/:id/open', requireStaff, async (req, res) => {
     let competition = competitionOf(req, res)
     if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, notebookNeeds(competition))) return
     competition = getCompetition(competition.id)
     if (!competition) return fail(res, 404, 'not_found', tr('competitions.refusal.notFound'))
     if (competition.state !== 'draft') {
@@ -1591,7 +1693,7 @@ export function adminCompetitionRoutes(): Router {
     async (req, res) => {
       const competition = competitionOf(req, res)
       if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, notebookNeeds(competition))) return
       const submission = submissionOf(competition, req, res)
       if (!submission) return
       /*
@@ -1616,7 +1718,7 @@ export function adminCompetitionRoutes(): Router {
     async (req, res) => {
       const competition = competitionOf(req, res)
       if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, {})) return
       const submission = submissionOf(competition, req, res)
       if (!submission) return
       /*
@@ -1684,7 +1786,7 @@ export function adminCompetitionRoutes(): Router {
   router.post('/api/admin/competitions/:id/rescore', requireStaff, async (req, res) => {
     const competition = competitionOf(req, res)
     if (!competition) return
-    if (!await executionAvailable(competition, res)) return
+    if (!await executionAvailable(competition, res, {})) return
     // The pump counts: a submission qualifies if its answer IS ON DISK, and
     // that is a question for the files, not for the row's state.
     res.status(202).json({ queued: rescoreCompetition(competition.id) })
diff --git a/server/src/routes/admin-environments.ts b/server/src/routes/admin-environments.ts
index 1f3f3333..38d62cc5 100644
--- a/server/src/routes/admin-environments.ts
+++ b/server/src/routes/admin-environments.ts
@@ -50,7 +50,7 @@ import {
 } from '../environments.js'
 import { sessionsOnEnvironment } from '../db.js'
 import { canSeeRoom } from '../admin/access.js'
-import { gpuBusy, gpuDevices } from '../kernel/pool.js'
+import { gpuBusy, gpuDevices, roomGpuDevices, submissionGpuCards } from '../kernel/pool.js'
 import {
   ENVIRONMENT_NAME,
   type AdminErrorBody,
@@ -102,13 +102,23 @@ function listChanges(before: string | null, after: string): { added: string[]; r
  * On a machine without cards docker is not asked at all: there is no point
  * paying for `docker ps -a` on every screen open for a list known to be
  * empty.
+ *
+ * Only the rooms' pool is counted: a card the owner gave to competition
+ * submissions is no class's to take (kernel/pool.ts · roomGpuDevices), and
+ * counting it here would promise a seminar a card the start then refuses.
+ * For the same reason a card of the pool that a submission still holds (given
+ * back to classes mid-run) is not free yet, whether its container is up or the
+ * queue has only just bound the card to a run that is about to start
+ * (kernel/pool.ts · submissionGpuCards). `runners` says how many went to
+ * submissions, so "no slices" can say why.
  */
-async function gpuState(): Promise<{ total: number; free: number }> {
-  if (usingRuntimeBroker()) return { total: 0, free: 0 }
-  const devices = gpuDevices()
-  if (devices.length === 0) return { total: 0, free: 0 }
-  const busy = new Set(await gpuBusy())
-  return { total: devices.length, free: devices.filter((device) => !busy.has(device)).length }
+async function gpuState(): Promise<{ total: number; free: number; runners: number }> {
+  if (usingRuntimeBroker()) return { total: 0, free: 0, runners: 0 }
+  const devices = roomGpuDevices()
+  const runners = gpuDevices().length - devices.length
+  if (devices.length === 0) return { total: 0, free: 0, runners }
+  const busy = new Set([...(await gpuBusy()), ...(await submissionGpuCards())])
+  return { total: devices.length, free: devices.filter((device) => !busy.has(device)).length, runners }
 }
 
 /**
diff --git a/server/src/routes/admin-resources.ts b/server/src/routes/admin-resources.ts
index a10b8b5c..6b4a4bd6 100644
--- a/server/src/routes/admin-resources.ts
+++ b/server/src/routes/admin-resources.ts
@@ -25,10 +25,11 @@ import {
   type MachineBounds,
 } from '../admin/resource-settings.js'
 import { slotsResolution } from '../competitions/capacity.js'
+import { runnerGpuDevices } from '../competitions/gpu.js'
 import { competitionDefaults } from '../competitions/settings.js'
 import { listNames } from '../environments.js'
 import { idleSweepSettingsChanged } from '../kernel/index.js'
-import { reapplyInstanceDefaults, runningKernelLimits, type RunningKernel } from '../kernel/pool.js'
+import { gpuBusy, gpuDevices, reapplyInstanceDefaults, runningKernelLimits, submissionGpuCards, type RunningKernel } from '../kernel/pool.js'
 import {
   cpuBounds,
   forgetResources,
@@ -83,6 +84,38 @@ function competitionShare(): ResourceBudget['competitions'] {
   }
 }
 
+/**
+ * ADAPTER, the same way: the machine's cards — all of KERNEL_GPUS, the ones a
+ * room's container holds now, and the submissions' (a promise, like the slots
+ * above, whether or not a submission runs). Each card is counted once. A card
+ * given to submissions that a class still holds counts as the submissions':
+ * it is theirs as soon as the class lets go. A card given back to classes
+ * that a submission still holds (its container, or the queue's binding of a
+ * run about to start) counts as the submissions' too, until that run ends: no
+ * class can take it before then (kernel/pool.ts · submissionGpuCards). Under
+ * Kubernetes the cluster has the cards, and the bar leaves them out.
+ *
+ * The census is the panel's, not the allocator's: an unanswered `docker ps`
+ * reads as "nobody holds a card", which `complete` on the memory side
+ * already warns about in the same answer.
+ */
+async function gpuShare(): Promise {
+  if (kernelBackend() === 'broker') return null
+  const devices = gpuDevices()
+  if (devices.length === 0) return { total: 0, rooms: 0, runners: 0 }
+  const [held, running] = await Promise.all([
+    gpuBusy().catch(() => [] as string[]),
+    submissionGpuCards().catch(() => [] as string[]),
+  ])
+  const submissions = new Set([...runnerGpuDevices(), ...running])
+  const rooms = new Set(held)
+  return {
+    total: devices.length,
+    rooms: devices.filter((device) => !submissions.has(device) && rooms.has(device)).length,
+    runners: devices.filter((device) => submissions.has(device)).length,
+  }
+}
+
 /* ---------------------------------------------------------------- budget */
 
 /**
@@ -128,6 +161,7 @@ async function budget(machine: InstanceResources): Promise {
     freeMb: Math.max(0, totalMb - reserveMb - promised),
     overcommitted: promised > totalMb - reserveMb,
     complete,
+    gpus: await gpuShare(),
   }
 }
 
diff --git a/server/src/routes/admin-running.ts b/server/src/routes/admin-running.ts
index 326bdb08..a9429899 100644
--- a/server/src/routes/admin-running.ts
+++ b/server/src/routes/admin-running.ts
@@ -205,6 +205,8 @@ function runRows(viewer: StaffLike): RunningRun[] {
       limitSec: competition ? limitsFor(competition, row.kind).wallSeconds : null,
       reservedMb: competition ? runReservationMb(competition, row.kind) : 0,
       canStop: visible,
+      // The runner card the attempt holds, as the queue took it (QueueRow.gpu); none on the CPU or the cluster.
+      gpu: row.gpu,
     })
   }
   return out
diff --git a/server/src/routes/competitions.ts b/server/src/routes/competitions.ts
index 4d34b034..a95fc838 100644
--- a/server/src/routes/competitions.ts
+++ b/server/src/routes/competitions.ts
@@ -42,7 +42,8 @@ import { zonedClock } from '@shared/time-zone'
 import { currentStaff } from '../admin/auth.js'
 import { canSeeCompetition } from '../competitions/scope.js'
 import { instanceNextDayStart, instanceTimeZone } from '../time-zone.js'
-import { assertCompetitionCapability, competitionCapabilities } from '../competitions/capabilities.js'
+import { assertCompetitionCapability, competitionCapabilities, notebookNeeds } from '../competitions/capabilities.js'
+import { competitionGpuSliceMb, runnerCardsHere, runnerGpuDevices, runnerGpuPerDevice } from '../competitions/gpu.js'
 import { competitionRevision } from '../dependencies/store.js'
 import { readEnvironmentInventory } from '../environment-inventory.js'
 import { acceptPinnedSubmission, executionRevision, publicExecution } from '../dependencies/service.js'
@@ -641,7 +642,10 @@ export function competitionRoutes(inventory = readEnvironmentInventory): Router
       // where the metric code and the row split seed are stripped from the
       // competition.
       competition: publicCompetition(competition),
-      capabilities: await competitionCapabilities(competition.environment, competitionRevision(competition.id)?.imageDigest),
+      capabilities: await competitionCapabilities(competition.environment, competitionRevision(competition.id)?.imageDigest, notebookNeeds(competition)),
+      gpuSliceMb: await competitionGpuSliceMb(competition),
+      // One run per card on this machine's cards: the run has the whole card (EntrantCompetitionView.gpuWholeCard).
+      gpuWholeCard: competition.limits.gpu === true && runnerCardsHere() && runnerGpuDevices().length > 0 && runnerGpuPerDevice() === 1,
       ...filesOf(competition.id),
       entrants: summary.entrants,
       submissions: summary.submissions,
@@ -958,13 +962,14 @@ export function competitionRoutes(inventory = readEnvironmentInventory): Router
       res.once('close', release)
     }
     try {
-      await assertCompetitionCapability('execution', competition.environment, competitionRevision(competition.id)?.imageDigest)
+      await assertCompetitionCapability('execution', competition.environment, competitionRevision(competition.id)?.imageDigest, notebookNeeds(competition))
     } catch (error) {
       return refuse(res, 503, 'unavailable', error instanceof Error ? error.message : tr('runtime.brokerUnavailable'))
     }
 
     readNotebook(req, res, competition, async (fileName, body, bundleId) => {
-      const revision = await executionRevision(competition)
+      // Asked with the card again: the owner may have taken the last one back while the body arrived.
+      const revision = await executionRevision(competition, notebookNeeds(competition))
       /*
        * The acceptance and the notebook on disk commit together or not at
        * all. A replacement cancels the person's waiting submission inside the
diff --git a/shared/admin.ts b/shared/admin.ts
index 82943faf..0ce883b5 100644
--- a/shared/admin.ts
+++ b/shared/admin.ts
@@ -798,6 +798,17 @@ export interface ResourceBudget {
    * answer the census: the bar then shows less than is really promised.
    */
   complete: boolean
+  /**
+   * The machine's cards, for a bar of their own: all of `KERNEL_GPUS`, the
+   * ones a room's container holds right now, and the submissions' — the ones
+   * given to competition submissions (a promise like the competition slots,
+   * held whether or not a submission runs), plus one given back to classes
+   * that a submission still runs on. Each card is counted once: a card given
+   * to submissions that a class has not let go yet is a submissions' card.
+   * What is left is free for the next GPU class. `null` — the cluster has the
+   * cards (Kubernetes), and the panel does not know them.
+   */
+  gpus: { total: number; rooms: number; runners: number } | null
 }
 
 /** GET /api/admin/resources, and the answer to PUT. */
@@ -959,6 +970,12 @@ export interface RunningRun {
   reservedMb: number
   /** Stopped through the competitions door (POST /api/admin/competitions/queue/kill). */
   canStop: boolean
+  /**
+   * The runner card the attempt holds, e.g. "1"; `null` — a run on the CPU.
+   * The card's id is the machine's business, so it is shown for hidden runs
+   * too, like `RunningClass.gpu`.
+   */
+  gpu: string | null
 }
 
 /** A dependency preparation (a competition's package bundle) holding its reservation now. */
@@ -1464,14 +1481,16 @@ export interface EnvironmentsState extends EnvironmentAbilities {
   /** Legacy wire field, always false: shared production execution is disabled. */
   shared: boolean
   /**
-   * GPU slices: how many are listed in KERNEL_GPUS and how many are free now.
+   * GPU slices: how many rooms may take (KERNEL_GPUS without the cards the
+   * owner gave to competition submissions) and how many of those are free now.
    *
    * A slice is free when it is on no room's container — the server counts
    * this, because only it knows. `total: 0` is an ordinary install without
    * cards, not a breakage: the panel says so in words and does not alarm
-   * anyone.
+   * anyone. `runners` — the cards given to submissions, so "no slices" can
+   * say why when every card went there; absent from servers before them.
    */
-  gpus: { total: number; free: number }
+  gpus: { total: number; free: number; runners?: number }
 }
 
 export interface SaveEnvironmentRequest {
diff --git a/shared/capabilities.ts b/shared/capabilities.ts
index c40ce909..b85bb2e8 100644
--- a/shared/capabilities.ts
+++ b/shared/capabilities.ts
@@ -1,3 +1,17 @@
 /** Operational availability, independent of whether an action is permitted. */
 export interface RuntimeCapability { available: boolean; code: string; reason: string | null }
-export interface CompetitionCapabilities { execution: RuntimeCapability; preparation: RuntimeCapability }
+/**
+ * `gpu` — whether the executor can give a notebook a share of a card: the
+ * broker reports it in /v1/health once the cluster has a GPU resource for
+ * submissions. Optional, and read as "no" when absent: a broker from before it
+ * cannot run a GPU job, and saying otherwise would queue work it refuses.
+ *
+ * `gpuSlots` — how many GPU notebooks of this instance the cluster runs at
+ * once, as its operator says (RUNTIME_COMPETITION_GPU_SLOTS, 1 to
+ * COMPETITION_GPU_SLOTS_MAX): the queue keeps that many at the broker and lets
+ * CPU work go past the rest, as it does on this machine's own cards. `null` —
+ * the operator did not say; absent — a broker from before the setting. Either
+ * way the queue keeps one: a GPU notebook the cluster cannot place holds a
+ * general slot for the half a minute the broker waits on the scheduler.
+ */
+export interface CompetitionCapabilities { execution: RuntimeCapability; preparation: RuntimeCapability; gpu?: RuntimeCapability; gpuSlots?: number | null }
diff --git a/shared/competition-runtime.ts b/shared/competition-runtime.ts
index 9fca7628..7ab5f4c0 100644
--- a/shared/competition-runtime.ts
+++ b/shared/competition-runtime.ts
@@ -34,6 +34,15 @@ export interface CompetitionJobIntent {
    * such a run fails rather than quietly reading the open example.
    */
   sealedInputs?: true
+  /**
+   * The notebook gets a share of a card: the broker adds the cluster's GPU
+   * resource (RUNTIME_COMPETITION_GPU_RESOURCE) and the GPU nodes' placement
+   * to its Pod, and refuses the job when it has none. A flag, not a resource
+   * name or a count: those are the operator's, never the app's. Notebook jobs
+   * only (the metric holds the hidden answers); an older broker refuses the
+   * unknown key, so such a run fails rather than quietly training on the CPU.
+   */
+  gpu?: true
   limits: CompetitionJobLimits
 }
 export interface CompetitionJobProgress { phase: 'dependencies' | 'notebook' | 'score' | 'resolve' | 'verify'; cell: number; cells: number; outputBytes: number }
@@ -47,19 +56,44 @@ export interface CompetitionJobStatus {
   oomKilled: boolean
   progress: CompetitionJobProgress | null
   error: string | null
-  failure?: { code: 'unschedulable' | 'image_unavailable' | 'runtime_unavailable' | 'export_failed'; resource?: 'memory' | 'cpu' | 'storage' | 'other' }
+  /** `resource: 'gpu'`: no node had a free unit of the GPU resource, or the node's device plugin could not hand one over. */
+  failure?: { code: 'unschedulable' | 'image_unavailable' | 'runtime_unavailable' | 'export_failed'; resource?: 'memory' | 'cpu' | 'storage' | 'gpu' | 'other' }
 }
 export interface CompetitionJobCollection { files: {name:string;bytes:number;sha256:string}[]; totalBytes:number }
 export const COMPETITION_JOB_ID = /^[a-f0-9]{32}$/
 export const COMPETITION_ATTEMPT_ID = /^[a-f0-9]{32}$/
 export const COMPETITION_ROW_ID = /^[a-z2-9]{8}$/
 export const COMPETITION_REVISION = /^sha256:[a-f0-9]{64}$/
+/**
+ * The broker's `error` for a job that found no room on the cluster (a
+ * scheduler or kubelet refusal), next to `failure.code: 'unschedulable'`. When
+ * the broker knows more (the API server refused a GPU notebook's Pod for a
+ * quota, a missing RuntimeClass or an admission policy), `error` says that
+ * instead, and the app shows it to the teacher while the job waits in its
+ * queue (broker-runner.ts · CompetitionResourcePending).
+ */
+export const COMPETITION_RESOURCES_UNAVAILABLE = 'Competition resources are unavailable'
+/**
+ * RUNTIME_COMPETITION_GPU_SLOTS, the most GPU notebooks of one instance the
+ * operator says the cluster runs at once, is 1 to this; the broker reports it
+ * in /v1/health (CompetitionCapabilities.gpuSlots), and the app believes no
+ * other number.
+ */
+export const COMPETITION_GPU_SLOTS_MAX = 256
+/**
+ * RUNTIME_COMPETITION_GPU_PULL_SECONDS, how long the broker lets a GPU
+ * environment's image (10 to 13 GB) be pulled onto a node before a job's own
+ * time starts, is 0 to this. The app waits that long on top for such a job to
+ * leave the queue before it gives up on the broker, whatever the broker's own
+ * setting (broker-runner.ts · execute).
+ */
+export const COMPETITION_GPU_PULL_SECONDS_MAX = 3600
 const kinds: ReadonlySet = new Set(['notebook','metric','inventory','resolve','verify'])
 const integer=(v:unknown,min:number,max:number):v is number => Number.isSafeInteger(v)&&Number(v)>=min&&Number(v)<=max
 export function parseCompetitionJobIntent(value:unknown):CompetitionJobIntent {
  if(!value||typeof value!=='object'||Array.isArray(value))throw new Error('Invalid competition job')
  const v=value as Record
- if(Object.keys(v).some(key=>!['schemaVersion','jobId','kind','environment','revision','competitionId','submissionId','attemptId','preparationId','bundleId','sealedInputs','limits'].includes(key))||v.schemaVersion!==1||typeof v.kind!=='string'||!kinds.has(v.kind as CompetitionJobKind)||typeof v.jobId!=='string'||!COMPETITION_JOB_ID.test(v.jobId)||typeof v.environment!=='string'||!/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,30}$/.test(v.environment)||typeof v.revision!=='string'||!COMPETITION_REVISION.test(v.revision))throw new Error('Invalid competition job identity')
+ if(Object.keys(v).some(key=>!['schemaVersion','jobId','kind','environment','revision','competitionId','submissionId','attemptId','preparationId','bundleId','sealedInputs','gpu','limits'].includes(key))||v.schemaVersion!==1||typeof v.kind!=='string'||!kinds.has(v.kind as CompetitionJobKind)||typeof v.jobId!=='string'||!COMPETITION_JOB_ID.test(v.jobId)||typeof v.environment!=='string'||!/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,30}$/.test(v.environment)||typeof v.revision!=='string'||!COMPETITION_REVISION.test(v.revision))throw new Error('Invalid competition job identity')
  const present=(key:string,re:RegExp):boolean=>typeof v[key]==='string'&&re.test(v[key] as string)
  if((v.kind==='notebook'||v.kind==='metric')&&(!present('competitionId',COMPETITION_ROW_ID)||!present('submissionId',COMPETITION_ROW_ID)||!present('attemptId',COMPETITION_ATTEMPT_ID)))throw new Error('Invalid submission job identity')
  if((v.kind==='resolve'||v.kind==='verify')&&!present('preparationId',COMPETITION_ATTEMPT_ID))throw new Error('Invalid preparation job identity')
@@ -72,6 +106,8 @@ export function parseCompetitionJobIntent(value:unknown):CompetitionJobIntent {
  if(n.packagesMb!==undefined&&(v.kind!=='notebook'||v.bundleId===undefined))throw new Error('Unexpected package room')
  // The hidden test is read by a notebook and by nothing else: the metric never sees it.
  if(v.sealedInputs!==undefined&&(v.sealedInputs!==true||v.kind!=='notebook'))throw new Error('Unexpected sealed inputs')
+ // A card goes to a notebook and to nothing else: never the metric, which holds the answers.
+ if(v.gpu!==undefined&&(v.gpu!==true||v.kind!=='notebook'))throw new Error('Unexpected GPU request')
  if(v.kind==='notebook'||v.kind==='metric'){
   if(v.preparationId!==undefined||v.kind==='metric'&&v.bundleId!==undefined)throw new Error('Unexpected job reference')
  }else if(v.kind==='resolve'||v.kind==='verify'){
@@ -88,7 +124,7 @@ export function parseCompetitionJobStatus(value:unknown):CompetitionJobStatus {
  if(Object.keys(v).some(k=>!['jobId','kind','phase','startedAt','finishedAt','exitCode','oomKilled','progress','error','failure'].includes(k))||typeof v.jobId!=='string'||!COMPETITION_JOB_ID.test(v.jobId)||typeof v.kind!=='string'||!kinds.has(v.kind as CompetitionJobKind)||typeof v.phase!=='string'||!phases.has(v.phase as CompetitionJobStatus['phase'])||!integer(v.startedAt,0,Number.MAX_SAFE_INTEGER)||v.finishedAt!==null&&!integer(v.finishedAt,0,Number.MAX_SAFE_INTEGER)||v.exitCode!==null&&!integer(v.exitCode,0,255)||typeof v.oomKilled!=='boolean'||v.error!==null&&(typeof v.error!=='string'||v.error.length>500))throw new Error('Invalid competition job status')
  if(v.failure!==undefined){
   const f=v.failure
-  if(!f||typeof f!=='object'||Array.isArray(f)||Object.keys(f).some(k=>!['code','resource'].includes(k))||!['unschedulable','image_unavailable','runtime_unavailable','export_failed'].includes((f as Record).code as string)||((f as Record).resource!==undefined&&!['memory','cpu','storage','other'].includes((f as Record).resource as string)))throw new Error('Invalid competition job failure')
+  if(!f||typeof f!=='object'||Array.isArray(f)||Object.keys(f).some(k=>!['code','resource'].includes(k))||!['unschedulable','image_unavailable','runtime_unavailable','export_failed'].includes((f as Record).code as string)||((f as Record).resource!==undefined&&!['memory','cpu','storage','gpu','other'].includes((f as Record).resource as string)))throw new Error('Invalid competition job failure')
  }
  if(v.progress!==null){
   if(!v.progress||typeof v.progress!=='object'||Array.isArray(v.progress))throw new Error('Invalid competition job progress')
diff --git a/shared/competitions-api.ts b/shared/competitions-api.ts
index 38874673..2f6d40f0 100644
--- a/shared/competitions-api.ts
+++ b/shared/competitions-api.ts
@@ -142,6 +142,13 @@ export interface RunningNow {
    * blank, and there is no kill button for it.
    */
   hidden?: boolean
+  /**
+   * The runner card the run holds and the video memory it is promised there —
+   * the share it was taken at, whatever the runs per card are now (null — the
+   * card did not say its size, so nothing is promised); `null` — a run on the
+   * CPU. The card is held until the attempt ends, metric included.
+   */
+  gpu: { device: string; sliceMb: number | null } | null
 }
 
 /** A row of the "WAITING · N" list. */
@@ -159,6 +166,13 @@ export interface WaitingRow {
   baseline: boolean
   /** A late submission (outside the standings); the queue takes on-time work first. */
   late?: boolean
+  /**
+   * The job needs a share of a runner card: a notebook of a competition with
+   * the card on. With every share taken it is passed over while CPU work
+   * flows past it, and the console says what it waits for
+   * (admin/competitions.ts · waitsForCard). Absent from servers before cards.
+   */
+  gpu?: boolean
 }
 
 /**
@@ -179,6 +193,19 @@ export interface QueueSnapshot {
   doneToday: number
   /** "2 min 40 s on average"; null — nothing has finished today yet. */
   averageMs: number | null
+  /**
+   * The runner cards, when any are given to submissions: how many GPU runs
+   * may go at once (cards × runs per card, less the cards a class still
+   * holds), how many of those places are taken now, and the video memory each
+   * is promised (null — unknown). A run taken when fewer runs shared a card
+   * holds the places of its bigger share until it ends, so `busy` may exceed
+   * the runs on the cards. A GPU run needs a general slot AND a share of a
+   * card, so `slots` above stays the ceiling for both kinds. With no card
+   * given while GPU notebooks wait, zero places (`slots: 0`): they wait for a
+   * card nobody gave. `null` — no card is given and no GPU work waits, or the
+   * cards are the cluster's.
+   */
+  gpu: { slots: number; busy: number; sliceMb: number | null } | null
 }
 
 export interface CompetitionsList {
@@ -238,6 +265,14 @@ export interface SplitView {
 export interface CompetitionView {
   capabilities?: CompetitionCapabilities
   competition: Competition
+  /**
+   * The video memory each run of this competition is promised, MB, next to
+   * its limits: a runner card's memory over the runs sharing it, the smallest
+   * card's. `null` — the competition runs on the CPU (`limits.gpu` off), or
+   * the size is unknown (no card given yet, a card that did not say, or a
+   * cluster that decides it). Never a number that is not enforced.
+   */
+  gpuSliceMb: number | null
   openFiles: FileView[]
   /** Hidden answers: names, rows and columns — the contents are never handed out. */
   hiddenFiles: FileView[]
diff --git a/shared/competitions-entrant.ts b/shared/competitions-entrant.ts
index 4eb58414..1f7c832a 100644
--- a/shared/competitions-entrant.ts
+++ b/shared/competitions-entrant.ts
@@ -125,6 +125,20 @@ export interface EntrantCompetitionList {
 export interface EntrantCompetitionView {
   capabilities?: CompetitionCapabilities
   competition: CompetitionPublic
+  /**
+   * The video memory a run gets, MB, beside the checking conditions; `null` —
+   * the competition runs on the CPU, or the size is not known and so not
+   * promised (competitions-api.ts · CompetitionView.gpuSliceMb).
+   */
+  gpuSliceMb: number | null
+  /**
+   * Each run has a card to itself (one run per runner card): the conditions
+   * say «вся карта» rather than a share — with one run per card the share is
+   * the whole card, and «срез ≈ 24 ГБ» would read as a quarter of something.
+   * Absent or false — a share (`gpuSliceMb`), the CPU, or the cluster's card,
+   * whose size the cluster decides.
+   */
+  gpuWholeCard?: boolean
   files: EntrantFile[]
   /** The hidden test, as names and rows; empty — the check reads the open files only. */
   sealedFiles?: EntrantSealedFile[]
diff --git a/shared/competitions-settings.ts b/shared/competitions-settings.ts
index 29457d87..a4679226 100644
--- a/shared/competitions-settings.ts
+++ b/shared/competitions-settings.ts
@@ -19,6 +19,14 @@ export const SETTINGS_LIMITS = {
   memoryMb: LIMITS.memoryMb,
   cpus: LIMITS.cpus,
   perDay: LIMITS.perDay,
+  /**
+   * Submissions that share one runner card at a time. One by default: a card
+   * time-shared by two notebooks is two notebooks each running at half speed
+   * with half the video memory, and that is a choice the owner makes, not a
+   * default. Eight is where a 24 GB card leaves each run less than its CUDA
+   * context and a model worth training.
+   */
+  gpuPerDevice: { min: 1, max: 8, default: 1 },
 } as const
 
 /** Where a number came from: the owner's choice, the environment, or our default. */
@@ -39,8 +47,84 @@ export interface SlotsResolution {
   source: SettingSource
 }
 
-/** The limits a new competition starts with, field by field. */
-export type CompetitionDefaults = CompetitionLimits
+/**
+ * The limits a new competition starts with, field by field.
+ *
+ * Without the card: a new competition is born on the `base` environment, which
+ * has no use for one, and whether a competition gets a card is the teacher's
+ * switch on an environment that needs it (CompetitionLimits.gpu), never a
+ * machine-wide default that would hand cards to competitions nobody looked at.
+ */
+export type CompetitionDefaults = Omit
+
+/**
+ * One entry of `KERNEL_GPUS` as the owner's form shows it.
+ *
+ * `holder` is who the card is for: `runners` — given to competition
+ * submissions, or given back to classes while a submission still runs on it;
+ * `rooms` — in the rooms' pool and held by a room's container right now;
+ * `free` — in the rooms' pool and held by nobody. `heldByRoom` says the
+ * room's container is on it whatever the holder: a card given to submissions
+ * while a class still had it stays with that class until its container ends,
+ * and submissions wait for it until then. `heldBySubmission` is the mirror:
+ * a submission's container is on it (by the `colloq.gpu` label), so a card
+ * given back to classes mid-run stays the submission's until the run ends,
+ * and no class can start on it meanwhile (kernel/pool.ts · takeGpu).
+ *
+ * `name` and `memoryMb` are what the card says about itself (nvidia-smi, from
+ * the server or from a throwaway container); `null` — it could not be asked,
+ * and the form then says the size is unknown rather than guessing.
+ */
+export interface CompetitionGpuCard {
+  id: string
+  name: string | null
+  memoryMb: number | null
+  holder: 'runners' | 'rooms' | 'free'
+  heldByRoom: boolean
+  heldBySubmission: boolean
+}
+
+/**
+ * The cards competition submissions run on — the «Видеокарты для посылок»
+ * block of the Resources tab.
+ */
+export interface CompetitionGpuSettings {
+  /** Every card of `KERNEL_GPUS`, in its order; empty under Kubernetes, where the cluster has them. */
+  cards: CompetitionGpuCard[]
+  /** The cards given to submissions in effect: the chosen ones that are in `KERNEL_GPUS`. */
+  devices: string[]
+  /** Submissions sharing one card at a time. */
+  perDevice: number
+  /**
+   * The video memory every GPU run is promised, MB: a card's memory divided
+   * by `perDevice`, the smallest over the given cards — whichever card a run
+   * lands on, it gets at least this. With one run per card it is the whole
+   * card. `null` — no card is given, or one of them could not say its size;
+   * then nothing is promised and nothing is enforced.
+   */
+  sliceMb: number | null
+  source: { devices: SettingSource; perDevice: SettingSource }
+  bounds: { perDevice: { min: number; max: number } }
+  /**
+   * Set by the cluster, not by this form: under Kubernetes the submissions'
+   * card is a resource of the cluster (RUNTIME_COMPETITION_GPU_RESOURCE), and
+   * the panel neither lists nor gives cards.
+   */
+  managed: boolean
+  /**
+   * What answers once a saved choice is forgotten (`gpuDevices: null`,
+   * `gpuPerDevice: null`): the cards COMPETITION_GPUS names that
+   * `KERNEL_GPUS` lists, and COMPETITION_GPU_PER_DEVICE — or, unset, the
+   * defaults: no card, one run per card. The owner's «вернуть …» shows it at
+   * once and names its source. Absent under the cluster, and from a server
+   * from before it.
+   */
+  fallback?: {
+    devices: string[]
+    perDevice: number
+    source: { devices: Exclude; perDevice: Exclude }
+  }
+}
 
 export interface CompetitionSettings {
   slots: SlotsResolution & {
@@ -66,6 +150,8 @@ export interface CompetitionSettings {
     cpus: { min: number; max: number }
     perDay: { min: number; max: number }
   }
+  /** The cards given to competition submissions. */
+  gpu: CompetitionGpuSettings
 }
 
 /**
@@ -81,3 +167,22 @@ export interface CompetitionSettingsInput {
   cpus?: number | null
   perDay?: number | null
 }
+
+/**
+ * The runner cards' half of the same form, on the same door.
+ *
+ * Its own interface, not two more fields above: everything above is a number
+ * the form reads from a text field, and a list of cards is not one — a form
+ * that walks the fields above as numbers stays right without learning about
+ * lists. The door takes both halves in one body
+ * (`CompetitionSettingsInput & CompetitionGpuSettingsInput`).
+ *
+ * `gpuDevices` — the `KERNEL_GPUS` entries given to submissions, `[]` to give
+ * none; `gpuPerDevice` — submissions per card. `null` forgets the saved value,
+ * and COMPETITION_GPUS / COMPETITION_GPU_PER_DEVICE or the default answer
+ * again.
+ */
+export interface CompetitionGpuSettingsInput {
+  gpuDevices?: string[] | null
+  gpuPerDevice?: number | null
+}
diff --git a/shared/competitions.ts b/shared/competitions.ts
index 15d5090f..c54a7eca 100644
--- a/shared/competitions.ts
+++ b/shared/competitions.ts
@@ -105,6 +105,24 @@ export interface CompetitionLimits {
   cpus: number
   /** Submissions per day per participant; 0 — no limit. */
   perDay: number
+  /**
+   * The notebook runs with a share of a video card — «Видеокарта» in the
+   * editor.
+   *
+   * A switch of its own rather than "the environment needs a GPU": a
+   * competition made on a GPU environment before runner cards existed ran its
+   * CUDA torch on the CPU, and turning a card on under it by itself would
+   * change how fast every one of its submissions runs, in the middle of the
+   * competition. So the teacher turns it on, and only on an environment that
+   * needs a GPU (competitions/panel.ts · gpuEnvironmentRefusal). Only the
+   * notebook step gets the card; the metric never does — it holds the hidden
+   * answers.
+   *
+   * Absent means false: everything made before the switch, and every fixture
+   * that never heard of it, runs on the CPU as it always did. The store always
+   * writes it out.
+   */
+  gpu?: boolean
 }
 
 export interface Competition {
diff --git a/shared/locales/admin.ts b/shared/locales/admin.ts
index 69723f8a..e5c86cad 100644
--- a/shared/locales/admin.ts
+++ b/shared/locales/admin.ts
@@ -585,6 +585,22 @@ export const adminMessages: MessageCatalog = {
     "ru": "В KERNEL_GPUS не настроены срезы GPU. Занятия с GPU-окружением не смогут запустить ядро, пока срез GPU не станет доступен.",
     "en": "No GPU slices are configured in KERNEL_GPUS. Classes using a GPU environment cannot start a kernel until a GPU slice is available."
   },
+  "admin.env.gpuAllToSubmissions": {
+    "ru": "Все видеокарты машины ({count}) отданы посылкам соревнований: занятия с GPU-окружением не запустят ядро, пока владелец не вернёт карту занятиям во вкладке «Ресурсы».",
+    "en": "Every GPU on this machine ({count}) is given to competition submissions: classes on a GPU environment cannot start a kernel until the owner gives a card back to classes on the Resources tab."
+  },
+  "admin.env.gpuRunnersToo": {
+    "ru": {
+      "one": "Ещё {count} карта отдана посылкам соревнований.",
+      "few": "Ещё {count} карты отданы посылкам соревнований.",
+      "many": "Ещё {count} карт отдано посылкам соревнований.",
+      "other": "Ещё {count} карты отданы посылкам соревнований."
+    },
+    "en": {
+      "one": "{count} more card is given to competition submissions.",
+      "other": "{count} more cards are given to competition submissions."
+    }
+  },
   "admin.a.room.on.this.environment.holds.a.gpu.slice.for.as.long.as.its.c": {
     "ru": "Комната с этим окружением занимает срез GPU, пока работает её контейнер",
     "en": "A room on this environment holds a GPU slice for as long as its container lives"
@@ -941,6 +957,14 @@ export const adminMessages: MessageCatalog = {
     "ru": "Все срезы заняты контейнерами других занятий, в том числе неактивных. Занятие можно создать, но для запуска ядра потребуется освободить срез.",
     "en": "All slices are held by containers from other classes, including inactive ones. You can create the class, but a slice must be freed before starting its kernel."
   },
+  "admin.gpu.allToSubmissions": {
+    "ru": "Все видеокарты отданы посылкам.",
+    "en": "Every GPU is given to submissions."
+  },
+  "admin.gpu.allToSubmissionsNote": {
+    "ru": "Окружению нужен GPU, а все видеокарты машины владелец отдал посылкам соревнований. Занятие можно создать, но ядро не запустится, пока владелец не вернёт карту занятиям во вкладке «Ресурсы».",
+    "en": "This environment needs a GPU, but the owner gave every card on this machine to competition submissions. You can create the class, but its kernel will not start until the owner gives a card back to classes on the Resources tab."
+  },
   "admin.materials": {
     "ru": "Материалы",
     "en": "Materials"
@@ -2638,6 +2662,10 @@ export const adminMessages: MessageCatalog = {
     "ru": "очередь приостановлена",
     "en": "the queue is paused"
   },
+  "admin.competitions.gpuBusyN": {
+    "ru": "видеокарты {busy} из {count}",
+    "en": "GPUs {busy} of {count}"
+  },
   "admin.competitions.queuePausedLong": {
     "ru": "Очередь приостановлена: посылки принимаются, но не исполняются.",
     "en": "The queue is paused: submissions are accepted but not run."
@@ -3274,6 +3302,98 @@ export const adminMessages: MessageCatalog = {
     "ru": "На этой машине свободно {free} ГБ: одна посылка за раз оставляет идущему занятию {rest} ГБ.",
     "en": "This machine has {free} GB free: one submission at a time leaves {rest} GB to the class in progress."
   },
+  "admin.competitions.limit.gpu": {
+    "ru": "ВИДЕОКАРТА",
+    "en": "GPU"
+  },
+  "admin.competitions.gpuOff": {
+    "ru": "нет",
+    "en": "off"
+  },
+  "admin.competitions.gpuSlice": {
+    "ru": "срез ≈\u00a0{value}\u00a0ГБ",
+    "en": "share ≈\u00a0{value}\u00a0GB"
+  },
+  "admin.competitions.gpuSliceBare": {
+    "ru": "срез",
+    "en": "share"
+  },
+  "admin.competitions.gpuWhole": {
+    "ru": "вся карта",
+    "en": "whole card"
+  },
+  "admin.competitions.gpuOnHint": {
+    "ru": "посылка видит карту как cuda:0; срез и число мест задаёт владелец в «Ресурсах»",
+    "en": "the submission sees the card as cuda:0; the owner sets the share and the number of places in Resources"
+  },
+  "admin.competitions.gpuOnHintCluster": {
+    "ru": "посылка видит карту как cuda:0; долю карты задаёт кластер",
+    "en": "the submission sees the card as cuda:0; the cluster decides its share"
+  },
+  "admin.competitions.gpuCpuEnv": {
+    "ru": "окружение {environment} без CUDA — карту посылке дать нельзя, считает процессор",
+    "en": "{environment} has no CUDA: a submission cannot get a card, the CPU computes"
+  },
+  "admin.competitions.gpuOffHint": {
+    "ru": "посылки пойдут на процессоре: torch с CUDA работает и без карты, но медленнее",
+    "en": "submissions will run on the CPU: CUDA torch works without a card, only slower"
+  },
+  "admin.competitions.gpuNoCards": {
+    "ru": "посылкам не выдано ни одной видеокарты",
+    "en": "no GPU is given to submissions"
+  },
+  "admin.competitions.gpuNoCardsOwner": {
+    "ru": "Пока карт нет, посылки не примутся, а сэмпл-тетрадь не проверится: участник увидит, что видеокарт для посылок нет. Владелец отдаёт карты посылкам во вкладке «Ресурсы».",
+    "en": "Until there is a card, submissions are not accepted and the sample notebook is not checked: the participant sees that there are no GPUs for submissions. The owner gives cards to submissions on the Resources tab."
+  },
+  "admin.competitions.gpuNoCardsTeacher": {
+    "ru": "Пока карт нет, посылки не примутся, а сэмпл-тетрадь не проверится: участник увидит, что видеокарт для посылок нет. Попросите владельца отдать посылкам карту во вкладке «Ресурсы».",
+    "en": "Until there is a card, submissions are not accepted and the sample notebook is not checked: the participant sees that there are no GPUs for submissions. Ask the owner to give submissions a card on the Resources tab."
+  },
+  "admin.competitions.gpuNoCardsClusterOwner": {
+    "ru": "Пока кластер не выдаёт посылкам видеокарт, посылки не примутся, а сэмпл-тетрадь не проверится: участник увидит, что видеокарт для посылок нет. Видеокарты посылкам выдаёт кластер: задайте gpu.competitions.resource в Helm (у брокера это RUNTIME_COMPETITION_GPU_RESOURCE) — или обновите брокер, если он старше приложения.",
+    "en": "Until the cluster gives submissions a GPU, submissions are not accepted and the sample notebook is not checked: the participant sees that there are no GPUs for submissions. The cluster gives GPUs to submissions: set gpu.competitions.resource in Helm (RUNTIME_COMPETITION_GPU_RESOURCE on the broker), or update the broker if it is older than the app."
+  },
+  "admin.competitions.openResources": {
+    "ru": "Открыть «Ресурсы» в новой вкладке",
+    "en": "Open Resources in a new tab"
+  },
+  "admin.competitions.gpuPlaces": {
+    "ru": {
+      "one": "На картах для посылок {count} место: {cards} по {per}, {share}.",
+      "few": "На картах для посылок {count} места: {cards} по {per}, {share}.",
+      "many": "На картах для посылок {count} мест: {cards} по {per}, {share}.",
+      "other": "На картах для посылок {count} места: {cards} по {per}, {share}."
+    },
+    "en": {
+      "one": "{count} place on the submission cards ({cards}, {per} each), {share}.",
+      "other": "{count} places on the submission cards ({cards}, {per} each), {share}."
+    }
+  },
+  "admin.competitions.gpuRunsPer": {
+    "ru": {
+      "one": "{count} посылке",
+      "few": "{count} посылки",
+      "many": "{count} посылок",
+      "other": "{count} посылки"
+    },
+    "en": {
+      "one": "{count} run",
+      "other": "{count} runs"
+    }
+  },
+  "admin.competitions.gpuShareWhole": {
+    "ru": "карта целиком",
+    "en": "the whole card"
+  },
+  "admin.competitions.gpuShareUnknown": {
+    "ru": "объём карты неизвестен",
+    "en": "card size unknown"
+  },
+  "admin.competitions.gpuMemoryNote": {
+    "ru": "Память {memory} ГБ — это память машины для данных и torch, видеопамять считается отдельно.",
+    "en": "Memory {memory} GB is the machine's memory for data and torch; GPU memory is counted separately."
+  },
   "admin.competitions.section.terms": {
     "ru": "Сроки и зачёт",
     "en": "Dates and standings"
@@ -3819,6 +3939,86 @@ export const adminMessages: MessageCatalog = {
     "ru": "{rooms} — комнаты, {own} — личные тетради, {queue} — исполнители соревнований. Потоки numpy и torch в каждом контейнере ограничены его ядрами, поэтому соседи не отнимают друг у друга машину.",
     "en": "{rooms} for rooms, {own} for personal notebooks, {queue} for competition executors. numpy and torch threads in each container are capped at its cores, so neighbours do not take the machine from each other."
   },
+  "admin.resourcesTab.gpus": {
+    "ru": "Видеокарты",
+    "en": "GPUs"
+  },
+  "admin.resourcesTab.gpusOnMachine": {
+    "ru": {
+      "one": "карта на машине",
+      "few": "карты на машине",
+      "many": "карт на машине",
+      "other": "карты на машине"
+    },
+    "en": {
+      "one": "card on the machine",
+      "other": "cards on the machine"
+    }
+  },
+  "admin.resourcesTab.gpusFree": {
+    "ru": {
+      "one": "свободна {count} карта",
+      "few": "свободны {count} карты",
+      "many": "свободно {count} карт",
+      "other": "свободно {count} карты"
+    },
+    "en": {
+      "one": "{count} card free",
+      "other": "{count} cards free"
+    }
+  },
+  "admin.resourcesTab.gpusBusyOne": {
+    "ru": "свободной нет — карта при деле",
+    "en": "none free — the card is in use"
+  },
+  "admin.resourcesTab.gpusBusyTwo": {
+    "ru": "свободных нет — обе при деле",
+    "en": "none free — both are in use"
+  },
+  "admin.resourcesTab.gpusBusyAll": {
+    "ru": "свободных нет — все при деле",
+    "en": "none free — all are in use"
+  },
+  "admin.resourcesTab.gpuClasses": {
+    "ru": "Классы",
+    "en": "Classes"
+  },
+  "admin.resourcesTab.gpuSubmissions": {
+    "ru": "Посылки",
+    "en": "Submissions"
+  },
+  "admin.resourcesTab.gpuCount": {
+    "ru": {
+      "one": "{count} карта",
+      "few": "{count} карты",
+      "many": "{count} карт",
+      "other": "{count} карты"
+    },
+    "en": {
+      "one": "{count} card",
+      "other": "{count} cards"
+    }
+  },
+  "admin.resourcesTab.gpuWholePerClass": {
+    "ru": "целиком на класс",
+    "en": "a whole card per class"
+  },
+  "admin.resourcesTab.gpuAtOnce": {
+    "ru": "по {count} сразу",
+    "en": "{count} at once"
+  },
+  "admin.resourcesTab.gpuWholePerRun": {
+    "ru": "каждой посылке целиком",
+    "en": "a whole card per submission"
+  },
+  "admin.resourcesTab.gpuSliceShort": {
+    "ru": "срез ≈\u00a0{value}\u00a0ГБ",
+    "en": "share ≈\u00a0{value}\u00a0GB"
+  },
+  "admin.resourcesTab.gpusNote": {
+    "ru": "Карта одна на двоих не бывает: класс и посылки никогда не делят видеопамять.",
+    "en": "A card is never shared between the two: a class and submissions never share GPU memory."
+  },
   "admin.resourcesTab.disk": {
     "ru": "Диск",
     "en": "Disk"
@@ -4476,8 +4676,8 @@ export const adminMessages: MessageCatalog = {
     "en": "Competitions"
   },
   "admin.resourcesTab.competitionsNote": {
-    "ru": "Каждая посылка исполняется с нуля в своём контейнере без сети. Здесь — сколько их идёт одновременно и что получает новое соревнование.",
-    "en": "Every submission runs from scratch in its own container without network. Here: how many run at once and what a new competition gets."
+    "ru": "Каждая посылка исполняется с нуля в своём контейнере без сети. Здесь — сколько их идёт одновременно, что получает новое соревнование и какие видеокарты отданы посылкам.",
+    "en": "Every submission runs from scratch in its own container without network. Here: how many run at once, what a new competition gets, and which GPUs are given to submissions."
   },
   "admin.resourcesTab.slots": {
     "ru": "Исполнителей одновременно",
@@ -4539,6 +4739,126 @@ export const adminMessages: MessageCatalog = {
     "ru": "Считается по участнику, а не по адресу: класс за одним школьным NAT больше не делит общий лимит.",
     "en": "Counted per participant, not per address: a class behind one school NAT no longer shares one limit."
   },
+  "admin.resourcesTab.gpuTitle": {
+    "ru": "Видеокарты для посылок",
+    "en": "GPUs for submissions"
+  },
+  "admin.resourcesTab.gpuCardsHere": {
+    "ru": {
+      "one": "на машине {count} карта",
+      "few": "на машине {count} карты",
+      "many": "на машине {count} карт",
+      "other": "на машине {count} карты"
+    },
+    "en": {
+      "one": "{count} card on the machine",
+      "other": "{count} cards on the machine"
+    }
+  },
+  "admin.resourcesTab.gpuSizeUnknown": {
+    "ru": "объём неизвестен",
+    "en": "size unknown"
+  },
+  "admin.resourcesTab.gpuHolder.runners": {
+    "ru": "у посылок",
+    "en": "for submissions"
+  },
+  "admin.resourcesTab.gpuHolder.leaving": {
+    "ru": "у класса, пока он открыт",
+    "en": "a class has it until it ends"
+  },
+  "admin.resourcesTab.gpuHolder.rooms": {
+    "ru": "у классов",
+    "en": "for classes"
+  },
+  "admin.resourcesTab.gpuHolder.finishing": {
+    "ru": "у посылки, пока она не досчитается",
+    "en": "a submission has it until its run ends"
+  },
+  "admin.resourcesTab.gpuHolder.free": {
+    "ru": "свободна",
+    "en": "free"
+  },
+  "admin.resourcesTab.gpuPick": {
+    "ru": "Отдать посылкам {card}",
+    "en": "Give {card} to submissions"
+  },
+  "admin.resourcesTab.gpuPerDevice": {
+    "ru": "Посылок на одной карте",
+    "en": "Runs per card"
+  },
+  "admin.resourcesTab.gpuPerDeviceUnit": {
+    "ru": "от {min} до {max}",
+    "en": "{min} to {max}"
+  },
+  "admin.resourcesTab.gpuSlice": {
+    "ru": "Срез",
+    "en": "Share"
+  },
+  "admin.resourcesTab.gpuSliceGb": {
+    "ru": "≈\u00a0{value}\u00a0ГБ",
+    "en": "≈\u00a0{value}\u00a0GB"
+  },
+  "admin.resourcesTab.gpuSliceWhole": {
+    "ru": "вся карта",
+    "en": "whole card"
+  },
+  "admin.resourcesTab.gpuSlicePerRun": {
+    "ru": "видеопамяти на посылку",
+    "en": "of GPU memory per submission"
+  },
+  "admin.resourcesTab.gpuPlaces": {
+    "ru": {
+      "one": "{count} место на картах",
+      "few": "{count} места на картах",
+      "many": "{count} мест на картах",
+      "other": "{count} места на картах"
+    },
+    "en": {
+      "one": "{count} place on the cards",
+      "other": "{count} places on the cards"
+    }
+  },
+  "admin.resourcesTab.gpuHeld": {
+    "ru": "Срез держится честно: torch в посылке не возьмёт больше {value} ГБ, а посылку, которая всё же заняла больше, сервер остановит и скажет участнику, что она вышла за свой срез.",
+    "en": "The share is enforced: torch in a submission will not take more than {value} GB, and a submission that takes more anyway is stopped by the server, which tells the participant it went past its share."
+  },
+  "admin.resourcesTab.gpuTimeShared": {
+    "ru": {
+      "one": "Карта делится по времени: срез бережёт память соседей, а не скорость — {count} посылка на одной карте считает медленнее, чем одна. Отданная посылкам карта уходит у классов; при 1 на карте каждая посылка получает её целиком.",
+      "few": "Карта делится по времени: срез бережёт память соседей, а не скорость — {count} посылки на одной карте считают медленнее, чем одна. Отданная посылкам карта уходит у классов; при 1 на карте каждая посылка получает её целиком.",
+      "many": "Карта делится по времени: срез бережёт память соседей, а не скорость — {count} посылок на одной карте считают медленнее, чем одна. Отданная посылкам карта уходит у классов; при 1 на карте каждая посылка получает её целиком.",
+      "other": "Карта делится по времени: срез бережёт память соседей, а не скорость — {count} посылки на одной карте считают медленнее, чем одна. Отданная посылкам карта уходит у классов; при 1 на карте каждая посылка получает её целиком."
+    },
+    "en": {
+      "one": "A card is shared in time: the share protects the neighbours' memory, not their speed — {count} submission on one card runs slower than one. A card given to submissions leaves the classes; at 1 per card each submission gets the whole card.",
+      "other": "A card is shared in time: the share protects the neighbours' memory, not their speed — {count} submissions on one card run slower than one. A card given to submissions leaves the classes; at 1 per card each submission gets the whole card."
+    }
+  },
+  "admin.resourcesTab.gpuWholeLine": {
+    "ru": "Каждая посылка получает карту целиком.",
+    "en": "Each submission gets the whole card."
+  },
+  "admin.resourcesTab.gpuLeavesClasses": {
+    "ru": "Отданная посылкам карта уходит у классов.",
+    "en": "A card given to submissions leaves the classes."
+  },
+  "admin.resourcesTab.gpuUnknownLine": {
+    "ru": "Объём карты неизвестен: доля видеопамяти не ограничена.",
+    "en": "The card's size is unknown: the GPU memory share is not capped."
+  },
+  "admin.resourcesTab.gpuNoneGiven": {
+    "ru": "Посылкам не отдана ни одна карта: соревнование с видеокартой не примет посылки.",
+    "en": "No card is given to submissions: a competition with a GPU will not accept submissions."
+  },
+  "admin.resourcesTab.gpuNoCards": {
+    "ru": "На этой машине видеокарт нет (KERNEL_GPUS пуст).",
+    "en": "This machine has no GPUs (KERNEL_GPUS is empty)."
+  },
+  "admin.resourcesTab.gpuCluster": {
+    "ru": "Видеокарты посылкам выдаёт кластер: gpu.competitions.resource в Helm.",
+    "en": "The cluster gives GPUs to submissions: gpu.competitions.resource in Helm."
+  },
   "admin.resourcesTab.filesTitle": {
     "ru": "Файлы",
     "en": "Files"
@@ -4711,6 +5031,10 @@ export const adminMessages: MessageCatalog = {
     "ru": "последняя досчитается ≈ через {eta}",
     "en": "the last result in ≈ {eta}"
   },
+  "admin.competitions.lastResultSoon": {
+    "ru": "последняя досчитается вот-вот",
+    "en": "the last result any moment now"
+  },
   "admin.competitions.medianShort": {
     "ru": "медиана {span}",
     "en": "median {span}"
@@ -4751,6 +5075,46 @@ export const adminMessages: MessageCatalog = {
       "other": "The queue is fair: whoever already has something running lets the others go first. Times assume {count} executors and this competition's median; the participant sees the same number."
     }
   },
+  "admin.competitions.slotsHeadGpu": {
+    "ru": "ИСПОЛНИТЕЛИ · {busy} ИЗ {count} · ВИДЕОКАРТЫ {gpuBusy} ИЗ {gpuCount}",
+    "en": "EXECUTORS · {busy} OF {count} · GPUS {gpuBusy} OF {gpuCount}"
+  },
+  "admin.competitions.waitingCard": {
+    "ru": {
+      "one": "{count} из них — видеокарту",
+      "few": "{count} из них — видеокарту",
+      "many": "{count} из них — видеокарту",
+      "other": "{count} из них — видеокарту"
+    },
+    "en": {
+      "one": "{count} of them for a GPU",
+      "other": "{count} of them for a GPU"
+    }
+  },
+  "admin.competitions.waitingCardOne": {
+    "ru": "одна из них — видеокарту",
+    "en": "one of them for a GPU"
+  },
+  "admin.competitions.gpuNoPlace": {
+    "ru": "Посылкам с видеокартой сейчас негде считать: ни одна карта не отдана посылкам или её ещё держит класс. Карту посылкам отдаёт владелец во вкладке «Ресурсы».",
+    "en": "GPU submissions have nowhere to run right now: no card is given to submissions, or a class still holds it. The owner gives submissions a card on the Resources tab."
+  },
+  "admin.competitions.waitingAllCard": {
+    "ru": {
+      "one": "{count} ждёт видеокарту",
+      "few": "{count} ждут видеокарту",
+      "many": "{count} ждут видеокарту",
+      "other": "{count} ждут видеокарту"
+    },
+    "en": {
+      "one": "{count} waiting for a GPU",
+      "other": "{count} waiting for a GPU"
+    }
+  },
+  "admin.competitions.waitsForCard": {
+    "ru": "ждёт место на видеокарте",
+    "en": "waiting for a place on a GPU"
+  },
   "admin.competitions.exampleBadge": {
     "ru": "пример",
     "en": "example"
diff --git a/shared/locales/competitions.ts b/shared/locales/competitions.ts
index 73cfd4bf..0c594780 100644
--- a/shared/locales/competitions.ts
+++ b/shared/locales/competitions.ts
@@ -519,6 +519,77 @@ export const competitionsMessages: MessageCatalog = {
     ru: 'Тетрадь заняла больше {count} ГБ и была остановлена на ячейке {cell}.',
     en: 'The notebook used more than {count} GB and was stopped at cell {cell}.',
   },
+  // A run with a share of a video card held more than the share and was
+  // stopped from outside (server/src/competitions/runner.ts · gpuKillNote).
+  // {peak} and {slice} are gigabytes to a tenth, in the instance's own
+  // decimal separator; all of it is ours, measured outside the notebook.
+  'competitions.answer.gpuOutOfMemory': {
+    ru: 'Посылка заняла на видеокарте {peak} ГБ при срезе {slice} ГБ; её остановили на ячейке {cell}, чтобы не мешать соседям по карте.',
+    en: 'The submission took {peak} GB on the GPU with a {slice} GB share; it was stopped at cell {cell} so it would not disturb the other runs on the card.',
+  },
+  // The same without the peak: the words a run on the hidden test gets under
+  // the brief output policy, where how far past its share a run went is a
+  // number its notebook chose (runner.ts · gpuKillNote).
+  'competitions.answer.gpuOverShare': {
+    ru: 'Посылка заняла на видеокарте больше своего среза в {slice} ГБ; её остановили на ячейке {cell}, чтобы не мешать соседям по карте.',
+    en: 'The submission took more than its {slice} GB share of the GPU; it was stopped at cell {cell} so it would not disturb the other runs on the card.',
+  },
+  // A run on a shared card whose video memory stopped being readable: the
+  // watchdog's samples failed {count} times in a row after the card had
+  // answered, and the run was stopped rather than left unwatched
+  // (server/src/competitions/docker-runner.ts · unreadCardNotes). It follows
+  // the "failed at cell" head in place of a traceback; no number in it is the
+  // notebook's, so a blind participant may read it too.
+  'competitions.answer.gpuUnread': {
+    ru: {
+      one: 'Посылку остановили снаружи: {count} раз подряд не удалось прочитать, сколько видеопамяти она занимает, а на карте, которую делят несколько посылок, прогон без присмотра не оставляют. Так бывает, когда тетрадь занимает процессор сотнями потоков или процессов, или когда сама карта перестаёт отвечать.',
+      few: 'Посылку остановили снаружи: {count} раза подряд не удалось прочитать, сколько видеопамяти она занимает, а на карте, которую делят несколько посылок, прогон без присмотра не оставляют. Так бывает, когда тетрадь занимает процессор сотнями потоков или процессов, или когда сама карта перестаёт отвечать.',
+      many: 'Посылку остановили снаружи: {count} раз подряд не удалось прочитать, сколько видеопамяти она занимает, а на карте, которую делят несколько посылок, прогон без присмотра не оставляют. Так бывает, когда тетрадь занимает процессор сотнями потоков или процессов, или когда сама карта перестаёт отвечать.',
+      other: 'Посылку остановили снаружи: {count} раза подряд не удалось прочитать, сколько видеопамяти она занимает, а на карте, которую делят несколько посылок, прогон без присмотра не оставляют. Так бывает, когда тетрадь занимает процессор сотнями потоков или процессов, или когда сама карта перестаёт отвечать.',
+    },
+    en: {
+      one: 'The submission was stopped from outside: how much GPU memory it held could not be read {count} time in a row, and a run nobody can watch is not left on a card shared with other runs. That happens when a notebook keeps its processor busy with hundreds of threads or processes, or when the card itself stops answering.',
+      other: 'The submission was stopped from outside: how much GPU memory it held could not be read {count} times in a row, and a run nobody can watch is not left on a card shared with other runs. That happens when a notebook keeps its processor busy with hundreds of threads or processes, or when the card itself stops answering.',
+    },
+  },
+  // The sentence after torch's own OutOfMemoryError in a run that had a card
+  // (runner.ts · gpuMemoryHint, Paper 15 · G4 #6): torch met its share inside
+  // the participant's cell, and the traceback says "CUDA out of memory" and
+  // nothing about the share. Ours, and so safe for a blind participant too.
+  // {slice} is gigabytes to a tenth; with a card of its own there is no number.
+  'competitions.answer.gpuOomShare': {
+    ru: 'У посылки срез видеокарты {slice} ГБ: уменьшите batch_size или модель.',
+    en: 'This submission has a {slice} GB GPU share: reduce the batch size or the model.',
+  },
+  'competitions.answer.gpuOomWhole': {
+    ru: 'У посылки вся карта: уменьшите batch_size или модель.',
+    en: 'This submission has the whole card: reduce the batch size or the model.',
+  },
+  // A share nobody could state: the cluster's card, or a card that did not say its size.
+  'competitions.answer.gpuOomCard': {
+    ru: 'Видеопамяти посылке не хватило: уменьшите batch_size или модель.',
+    en: 'The submission ran out of GPU memory: reduce the batch size or the model.',
+  },
+  // A run that ran out of video memory while the watchdog was stopping its
+  // neighbour on the same card for going past its share: put back into the
+  // queue once instead of being judged (runner.ts · neighbourOverran). Our
+  // own words with no number in them, so a blind participant reads them too.
+  'competitions.answer.gpuNeighbourRetry': {
+    ru: 'Соседняя посылка на той же видеокарте заняла больше своего среза, и этой посылке не хватило видеопамяти. Тетрадь тут ни при чём: посылка снова в очереди и запустится заново, эта попытка не засчитывается.',
+    en: 'Another submission on the same GPU took more than its share, and this one ran out of video memory. That is not the notebook’s fault: the submission is back in the queue and will run again; this attempt does not count.',
+  },
+  // The same, in the teacher's column. {device} is the card as KERNEL_GPUS names it.
+  'competitions.answer.gpuNeighbourRetryTeacher': {
+    ru: 'Видеокарта {device}: CUDA out of memory, пока сторож останавливал соседнюю посылку на этой карте за выход за срез. Прогон повторяется один раз.',
+    en: 'GPU {device}: CUDA out of memory while the watchdog was stopping a neighbouring submission on this card for going past its share. The run is repeated once.',
+  },
+  // The teacher's column of a GPU notebook whose card the owner gave back to
+  // classes between the queue's take and the start (runner.ts · runGpu): it
+  // never started there, and waits in the queue for a card still given.
+  'competitions.answer.gpuWithdrawnTeacher': {
+    ru: 'Видеокарту {device} вернули занятиям, пока прогон готовился к запуску: на ней он не начался и ждёт в очереди карту, отданную посылкам.',
+    en: 'GPU {device} was given back to classes while the run was getting ready to start: it did not start there, and waits in the queue for a card given to submissions.',
+  },
   'competitions.answer.kernelDied': {
     ru: 'Ядро умерло на ячейке {cell}. Так бывает от os._exit(), от падения в си и от нехватки памяти внутри самой ячейки.',
     en: 'The kernel died at cell {cell}. That happens with os._exit(), with a crash inside a C extension, and when a single cell runs out of memory.',
@@ -666,6 +737,53 @@ export const competitionsMessages: MessageCatalog = {
     ru: 'Посылку снял сам участник.',
     en: 'The participant cancelled the submission.',
   },
+  // A run with a share of a video card, in the teacher's column
+  // (server/src/competitions/runner.ts · cardNote). {device} is the card as
+  // KERNEL_GPUS names it; the numbers are megabytes as nvidia-smi reports
+  // them: {samples} the two readings that decided a kill, {limit} the line
+  // they crossed (the share plus its tolerance).
+  'competitions.answer.gpuKilled': {
+    ru: 'Видеокарта {device}: два замера подряд выше {limit} МБ ({samples} МБ) при срезе {slice} МБ — прогон остановлен снаружи.',
+    en: 'GPU {device}: two samples in a row above {limit} MB ({samples} MB) with a {slice} MB share — the run was stopped from outside.',
+  },
+  'competitions.answer.gpuPeak': {
+    ru: 'Видеокарта {device}: пик {peak} МБ при срезе {slice} МБ.',
+    en: 'GPU {device}: peak {peak} MB with a {slice} MB share.',
+  },
+  // A watched run no sample ever saw holding memory: not "peak 0 MB", which
+  // read as a notebook that never touched the card (runner.ts · cardNote).
+  'competitions.answer.gpuPeakUnmeasured': {
+    ru: 'Видеокарта {device}: пик не измерен — ни один замер не увидел памяти посылки (срез {slice} МБ).',
+    en: 'GPU {device}: the peak was not measured — no sample saw the submission’s memory ({slice} MB share).',
+  },
+  // A run on a shared card stopped because its card could not be read
+  // (server/src/competitions/docker-runner.ts · unreadCardNotes), at the head
+  // of the teacher's log. {reason} is what the last failed sample said —
+  // docker's or nvidia-smi's own line, or gpuSampleStalled — and {slice} is
+  // megabytes, like gpuKilled.
+  'competitions.answer.gpuUnreadKilled': {
+    ru: {
+      one: 'Видеокарта {device}: {count} замер подряд не ответил (последний: {reason}) при срезе {slice} МБ — прогон остановлен снаружи: на общей карте прогон без присмотра не оставляют.',
+      few: 'Видеокарта {device}: {count} замера подряд не ответили (последний: {reason}) при срезе {slice} МБ — прогон остановлен снаружи: на общей карте прогон без присмотра не оставляют.',
+      many: 'Видеокарта {device}: {count} замеров подряд не ответили (последний: {reason}) при срезе {slice} МБ — прогон остановлен снаружи: на общей карте прогон без присмотра не оставляют.',
+      other: 'Видеокарта {device}: {count} замера подряд не ответили (последний: {reason}) при срезе {slice} МБ — прогон остановлен снаружи: на общей карте прогон без присмотра не оставляют.',
+    },
+    en: {
+      one: 'GPU {device}: {count} sample in a row did not answer (the last: {reason}) with a {slice} MB share — the run was stopped from outside: an unwatched run is not left on a shared card.',
+      other: 'GPU {device}: {count} samples in a row did not answer (the last: {reason}) with a {slice} MB share — the run was stopped from outside: an unwatched run is not left on a shared card.',
+    },
+  },
+  // Added when not one sample answered in the whole run: then it is the host,
+  // not the notebook, and the run ends as our own failure (harness_error).
+  'competitions.answer.gpuNeverRead': {
+    ru: 'Карта не ответила за весь прогон ни разу: если так кончается каждый прогон на ней, проверьте, работает ли nvidia-smi внутри контейнера на этом хосте.',
+    en: 'The card never answered during this run: if every run on it ends like this, check that nvidia-smi works inside a container on this host.',
+  },
+  // A sample that said nothing before its deadline ({seconds}), as {reason} above.
+  'competitions.answer.gpuSampleStalled': {
+    ru: 'нет ответа за {seconds} с',
+    en: 'no answer within {seconds} s',
+  },
   // The teacher's column of a waiting submission a newer upload replaced.
   'competitions.answer.replaced': {
     ru: 'Заменена посылкой #{number}, пока ждала в очереди.',
@@ -1162,6 +1280,17 @@ export const competitionsMessages: MessageCatalog = {
   'competitions.p.condData': { ru: 'Данные', en: 'Data' },
   'competitions.p.condDataValue': { ru: 'data/ · только чтение', en: 'data/ · read-only' },
   'competitions.p.condAnswer': { ru: 'Файл с прогнозами', en: 'Predictions file' },
+  // The card's row (Paper 15 · G4), only for a competition whose notebook runs
+  // with a share of a video card: the share the server holds, the whole card,
+  // or — where the size is the cluster's or unknown — only that there is one.
+  // Inside the container the card is always cuda:0.
+  'competitions.p.condGpu': { ru: 'Видеокарта', en: 'GPU' },
+  'competitions.p.condGpuSlice': { ru: 'срез ≈\u00a0{size}\u00a0ГБ · cuda:0', en: 'share ≈\u00a0{size}\u00a0GB · cuda:0' },
+  'competitions.p.condGpuWhole': { ru: 'вся карта · cuda:0', en: 'whole card · cuda:0' },
+  'competitions.p.condGpuOn': { ru: 'есть · cuda:0', en: 'yes · cuda:0' },
+  // No card is given to submissions yet: the send box refuses with the
+  // reason (runtime.gpuUnavailable), and this row says the same, not «есть».
+  'competitions.p.condGpuNone': { ru: 'посылкам пока не выдана', en: 'not given to submissions yet' },
 
   /* Лидерборд (P2 · мини-таблица, P3 · таблица) */
   'competitions.p.publicBoard': { ru: 'ПУБЛИЧНЫЙ ЛИДЕРБОРД', en: 'PUBLIC LEADERBOARD' },
@@ -1457,6 +1586,20 @@ export const competitionsMessages: MessageCatalog = {
       other: 'The notebook starts from scratch and runs every cell in order, in a container with no internet: up to {count} minutes, {memory} of memory.',
     },
   },
+  // The same first step for a notebook with a video card: {gpu} is the
+  // conditions row's own value («срез ≈ 6 ГБ · cuda:0»).
+  'competitions.p.howRunGpu': {
+    ru: {
+      one: 'Тетрадь запускается заново, все ячейки по порядку, в контейнере без интернета: до {count} минуты, {memory} памяти; видеокарта — {gpu}.',
+      few: 'Тетрадь запускается заново, все ячейки по порядку, в контейнере без интернета: до {count} минут, {memory} памяти; видеокарта — {gpu}.',
+      many: 'Тетрадь запускается заново, все ячейки по порядку, в контейнере без интернета: до {count} минут, {memory} памяти; видеокарта — {gpu}.',
+      other: 'Тетрадь запускается заново, все ячейки по порядку, в контейнере без интернета: до {count} минуты, {memory} памяти; видеокарта — {gpu}.',
+    },
+    en: {
+      one: 'The notebook starts from scratch and runs every cell in order, in a container with no internet: up to {count} minute, {memory} of memory; GPU: {gpu}.',
+      other: 'The notebook starts from scratch and runs every cell in order, in a container with no internet: up to {count} minutes, {memory} of memory; GPU: {gpu}.',
+    },
+  },
   'competitions.p.howSwap': {
     ru: 'В data/ лежат те же файлы, что выше, но на месте {file} — скрытый тест: те же столбцы, {rows}, другие строки. Читайте файл по этому пути и не рассчитывайте на число строк или конкретные id.',
     en: 'data/ holds the same files as above, but in place of {file} sits the hidden test: the same columns, {rows}, different rows. Read the file by this path and do not count on the number of rows or particular ids.',
@@ -1561,6 +1704,11 @@ export const competitionsMessages: MessageCatalog = {
     ru: 'Посылка этого участника ещё выполняется и не успела остановиться. Повторите через минуту.',
     en: 'One of this participant’s submissions is still running and has not stopped yet. Try again in a minute.',
   },
+  // Deleting a competition whose run did not stop in time (runner.ts · stopCompetitionRuns).
+  'competitions.refusal.competitionRunning': {
+    ru: 'Посылка этого соревнования ещё выполняется и не успела остановиться. Повторите удаление через минуту.',
+    en: 'A submission of this competition is still running and has not stopped yet. Try deleting it again in a minute.',
+  },
   'competitions.refusal.entrantBaseline': {
     ru: 'Это служебный участник сэмпл-тетради — его удалить нельзя.',
     en: 'This is the sample notebook’s service participant; it cannot be deleted.',
diff --git a/shared/locales/operations.ts b/shared/locales/operations.ts
index 8ea3f5da..ea3ed051 100644
--- a/shared/locales/operations.ts
+++ b/shared/locales/operations.ts
@@ -7,4 +7,6 @@ export const operationMessages: MessageCatalog = {
  'runtime.preparationVersion': {ru:'Для безопасной подготовки пакетов требуется Docker Engine 28 или новее.',en:'Safe package preparation requires Docker Engine 28 or newer.'},
  'runtime.storageUnavailable': {ru:'Хранилище исполнителя недоступно. Обратитесь к администратору.',en:'The executor storage is unavailable. Contact the administrator.'},
  'runtime.prepareUnavailable': {ru:'Подготовка пакетов недоступна.',en:'Package preparation is unavailable.'},
+ 'runtime.gpuUnavailable': {ru:'Этому соревнованию нужна видеокарта, а посылкам соревнований на этой машине не отдана ни одна. Владелец отдаёт видеокарты посылкам во вкладке «Ресурсы».',en:'This competition needs a GPU, but no card on this machine is given to competition submissions. The owner gives cards to submissions on the Resources tab.'},
+ 'runtime.gpuUnavailableCluster': {ru:'Этому соревнованию нужна видеокарта, а кластер не выдаёт видеокарт посылкам соревнований. Обратитесь к администратору.',en:'This competition needs a GPU, but the cluster gives no GPU to competition submissions. Contact the administrator.'},
 }
diff --git a/shared/locales/server.ts b/shared/locales/server.ts
index bf235ac8..07f195f4 100644
--- a/shared/locales/server.ts
+++ b/shared/locales/server.ts
@@ -1629,6 +1629,26 @@ export const serverMessages: MessageCatalog = {
     "ru": "Окружению «{p0}» нужен GPU, а свободных срезов нет: их {p1}, и все заняты другими занятиями. Подождите, пока освободится — срез уходит вместе с ядром комнаты, — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет.",
     "en": "Environment “{p0}” requires a GPU, but all {p1} available devices are in use by other classes. Wait for a room kernel to release one, or use a CPU environment. This environment cannot run on CPU alone."
   },
+  // Every card of KERNEL_GPUS was given to competition submissions (kernel/pool.ts · gpuRefusal).
+  "server.gpu.allToSubmissions": {
+    "ru": "Окружению «{p0}» нужен GPU, а все видеокарты этой машины ({p1}) отданы посылкам соревнований. Владелец может вернуть видеокарту занятиям во вкладке «Ресурсы» — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет.",
+    "en": "Environment “{p0}” requires a GPU, but every card on this machine ({p1}) is given to competition submissions. The owner can give a card back to classes on the Resources tab, or use a CPU environment. This environment cannot run on CPU alone."
+  },
+  // No free slice, and on {count} of them a competition submission is still
+  // running: the owner gave those cards back to classes mid-run, and a run
+  // keeps its card until it ends (kernel/pool.ts · gpuRefusal).
+  "server.gpu.submissionFinishing": {
+    "ru": {
+      "one": "Окружению «{p0}» нужен GPU, а свободных срезов нет: на {count} из них ещё досчитывается посылка соревнования — эту видеокарту вернули занятиям, пока посылка шла. Подождите, пока она закончится, — срез перейдёт занятиям сам, — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет.",
+      "few": "Окружению «{p0}» нужен GPU, а свободных срезов нет: на {count} из них ещё досчитываются посылки соревнований — эти видеокарты вернули занятиям, пока посылки шли. Подождите, пока одна из них закончится, — её срез перейдёт занятиям сам, — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет.",
+      "many": "Окружению «{p0}» нужен GPU, а свободных срезов нет: на {count} из них ещё досчитываются посылки соревнований — эти видеокарты вернули занятиям, пока посылки шли. Подождите, пока одна из них закончится, — её срез перейдёт занятиям сам, — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет.",
+      "other": "Окружению «{p0}» нужен GPU, а свободных срезов нет: на {count} из них ещё досчитываются посылки соревнований — эти видеокарты вернули занятиям, пока посылки шли. Подождите, пока одна из них закончится, — её срез перейдёт занятиям сам, — или откройте занятие на окружении без GPU: на процессоре это окружение не поедет."
+    },
+    "en": {
+      "one": "Environment “{p0}” requires a GPU, but no device is free: a competition submission is still running on {count} of them, a card given back to classes while the submission ran. Wait for it to end (the device then goes to classes by itself), or use a CPU environment. This environment cannot run on CPU alone.",
+      "other": "Environment “{p0}” requires a GPU, but no device is free: competition submissions are still running on {count} of them, cards given back to classes while the submissions ran. Wait for one of them to end (its device then goes to classes by itself), or use a CPU environment. This environment cannot run on CPU alone."
+    }
+  },
   "server.couldNotStartTheRoomContainer.cf7398": {
     "ru": "контейнер комнаты не удалось поднять: {p0}",
     "en": "could not start the room container: {p0}"
@@ -4172,5 +4192,18 @@ export const serverMessages: MessageCatalog = {
   "server.notes.importTooLong": {
     "ru": "Заметка к странице {p0} длиннее {p1} знаков ({p2}). Ничего не записали — сократите её и загрузите снова.",
     "en": "The note for page {p0} is longer than {p1} characters ({p2}). Nothing was written — shorten it and load again."
+  },
+  // Video cards for competition submissions (competitions/settings.ts, competitions/panel.ts).
+  "server.competitions.gpuUnknownDevice": {
+    "ru": "Видеокарты «{device}» нет в KERNEL_GPUS: посылкам можно отдать только карту из этого списка, записанную так же, как там.",
+    "en": "“{device}” is not in KERNEL_GPUS: only a card from that list, written the same way, can be given to submissions."
+  },
+  "server.competitions.gpuManaged": {
+    "ru": "На Kubernetes видеокарты для посылок соревнований задаёт кластер (RUNTIME_COMPETITION_GPU_RESOURCE), а не эта форма.",
+    "en": "On Kubernetes the cluster sets the cards for competition submissions (RUNTIME_COMPETITION_GPU_RESOURCE), not this form."
+  },
+  "server.competitions.gpuCpuEnvironment": {
+    "ru": "Видеокарта нужна только окружению с GPU, а «{environment}» считает на процессоре. Выберите окружение с GPU или выключите видеокарту.",
+    "en": "Only an environment with a GPU can use a card, and “{environment}” runs on the CPU. Pick a GPU environment or turn the card off."
   }
 }
diff --git a/site/docs/competitions.html b/site/docs/competitions.html
index a6c20f1a..be059807 100644
--- a/site/docs/competitions.html
+++ b/site/docs/competitions.html
@@ -29,14 +29,16 @@ 
    Публичная и приватная части
    Мет
 
 
    Скрытый тест
    Открытый data/test.csv класс скачивает, и по нему легко подогнать ответ: посчитать строки, выписать id. Скрытый тест — файл, который участник не скачивает: на каждой проверке тетрадь читает его по тому же пути, вместо открытого примера. Модель проверяется на строках, которых никто не видел.
    1. В редакторе, в разделе «Скрытый тест», загрузите файл своей кнопкой — не через открытые данные. В поле «Путь при проверке» укажите имя в data/: по умолчанию это имя самого файла. Файл с именем открытого примера подменяет пример, файл с новым именем добавляется к данным.
    2. Пример остаётся у участников: его скачивают и на нём отлаживают тетрадь. Сверьте колонки — редактор показывает, совпадают ли они с примером. Тетрадь, написанная под пример, упадёт на тесте с другими колонками.
    3. Ответы в solution.csv должны быть к строкам скрытого теста.
    4. Проверьте сэмпл-тетрадь заново: смена скрытого теста делает её прошлую проверку устаревшей.
    Каждый прогон получает собственный каталог data/: открытые файлы и скрытые на их месте. data/test.csv и /data/test.csv читают скрытый тест, метрика его не видит вовсе. Файл, заменённый посреди прогона, достаётся только следующим прогонам. Потолок — 200 МБ и 10 файлов, отдельно от открытых данных.
    Участник видит имя скрытого файла и число строк: в списке данных у примера стоит отметка «ПРИМЕР», а в условиях проверки — строка о скрытом тесте. Содержимого не отдаёт ни одна ссылка, ни участнику, ни вам: скачать скрытый тест нельзя.
    Пока тетрадь работает, скрытые строки у неё в памяти, и любой вывод может их вынести: test.head(), текст KeyError со значением, сообщение метрики, которое цитирует ответ. Поэтому в разделе «Что показывать участнику при проверке» два варианта:
    • «Только статус и тип ошибки» — по умолчанию. Участник видит исход, номер упавшей ячейки и тип исключения из короткого списка обычных (KeyError, ValueError…). Ни вывода ячеек, ни текста ошибки, ни трассировки; ссылка на тетрадь отдаёт тетрадь в том виде, в каком её прислали, а не исполненную. Отказ метрики пересказывается нашими словами, без значений, id и чисел из ответа.
    • «Полный вывод» — как без скрытого теста. Включайте, только если раскрыть эти строки не страшно: подогнать ответ по ним станет можно.
    Вы в ленте всегда видите весь вывод и исполненную тетрадь. Полный текст хранится при любом варианте, поэтому переключение действует сразу на все посылки, без повторного исполнения.
    
 
-
    Как исполняется посылка
    Участник нажимает «ВЫБРАТЬ ФАЙЛ» и отправляет тетрадь. Загрузки прямо из тетради занятия нет: только файл.
    Открытые файлы участник скачивает по одному или кнопкой «Скачать всё» над списком: в архиве папка data/ со всеми файлами и стартовые тетради рядом с ней, как при проверке, поэтому pd.read_csv('data/train.csv') работает сразу после распаковки. Скрытых ответов и скрытого теста в архиве нет.
    1. Посылка встаёт в очередь инстанса.
    2. В одноразовом контейнере без сети тетрадь исполняется целиком, ячейка за ячейкой, с нуля: сохранённых переменных нет, каталог data/ примонтирован только на чтение. Тетрадь обязана записать submission.csv.
    3. Файл забирается из контейнера, контейнер сносится.
    4. Во втором одноразовом контейнере — уже без данных участника и без его кода — запускается ваша метрика. Скрытые ответы бывают только здесь.
    Пределы соревнования (время на тетрадь, память, ядра) стоят в редакторе. По времени контейнер убивают снаружи, по памяти его ограничивает docker. Участник видит движение этапов вживую и, если тетрадь упала, свою трассировку целиком. У посылки есть ссылка «Скачать тетрадь с выводом»: это тетрадь в том виде, в каком её исполнил сервер, — у упавшей тоже, до ячейки, на которой она остановилась. Вывод в ней только этого прогона: присланный с ноутбука стирается перед запуском.
    ИсходЧто видит участник
    ГОТОВОПубличный результат и своё место.
    УПАЛА ТЕТРАДЬ«Упала на ячейке 5 из 14» и свою трассировку дословно.
    ЛИМИТ ВРЕМЕНИ«Не уложилась в 10 минут, ячейка 6 из 50».
    НЕ ХВАТИЛО ПАМЯТИ«Заняла больше 4 ГБ, ячейка 2».
    ОТВЕТ НЕ ПРИНЯТ«Дошла до конца, но submission.csv не оставила», «В ответе нет 3 строк, например id=17» — или текст вашего ParticipantVisibleError.
    УПАЛА МЕТРИКА«Проверяющий код упал» без подробностей; трассировку видите вы, и посылка ждёт пересчёта.
    Так выглядит соревнование без скрытого теста. Со скрытым тестом участник по умолчанию видит упавшую посылку коротко: исход, номер ячейки и тип исключения, без вывода и трассировки, см. скрытый тест.
    
+
    Как исполняется посылка
    Участник нажимает «ВЫБРАТЬ ФАЙЛ» и отправляет тетрадь. Загрузки прямо из тетради занятия нет: только файл.
    Открытые файлы участник скачивает по одному или кнопкой «Скачать всё» над списком: в архиве папка data/ со всеми файлами и стартовые тетради рядом с ней, как при проверке, поэтому pd.read_csv('data/train.csv') работает сразу после распаковки. Скрытых ответов и скрытого теста в архиве нет.
    1. Посылка встаёт в очередь инстанса.
    2. В одноразовом контейнере без сети тетрадь исполняется целиком, ячейка за ячейкой, с нуля: сохранённых переменных нет, каталог data/ примонтирован только на чтение. Тетрадь обязана записать submission.csv.
    3. Файл забирается из контейнера, контейнер сносится.
    4. Во втором одноразовом контейнере — уже без данных участника и без его кода — запускается ваша метрика. Скрытые ответы бывают только здесь.
    Пределы соревнования (время на тетрадь, память, ядра и переключатель «Видеокарта») стоят в редакторе. По времени контейнер убивают снаружи, по памяти его ограничивает docker. Участник видит движение этапов вживую и, если тетрадь упала, свою трассировку целиком. У посылки есть ссылка «Скачать тетрадь с выводом»: это тетрадь в том виде, в каком её исполнил сервер, — у упавшей тоже, до ячейки, на которой она остановилась. Вывод в ней только этого прогона: присланный с ноутбука стирается перед запуском.
    ИсходЧто видит участник
    ГОТОВОПубличный результат и своё место.
    УПАЛА ТЕТРАДЬ«Упала на ячейке 5 из 14» и свою трассировку дословно.
    ЛИМИТ ВРЕМЕНИ«Не уложилась в 10 минут, ячейка 6 из 50».
    НЕ ХВАТИЛО ПАМЯТИ«Заняла больше 4 ГБ, ячейка 2».
    ОТВЕТ НЕ ПРИНЯТ«Дошла до конца, но submission.csv не оставила», «В ответе нет 3 строк, например id=17» — или текст вашего ParticipantVisibleError.
    УПАЛА МЕТРИКА«Проверяющий код упал» без подробностей; трассировку видите вы, и посылка ждёт пересчёта.
    Так выглядит соревнование без скрытого теста. Со скрытым тестом участник по умолчанию видит упавшую посылку коротко: исход, номер ячейки и тип исключения, без вывода и трассировки, см. скрытый тест.
    
 
-
    Очередь и пульт
    Очередь одна на весь инстанс: она лежит в базе и переживает перезапуск сервера. Сколько посылок исполняется одновременно, по умолчанию считается по машине («Авто»): её ядра и память за вычетом 2 ядер и 2 ГБ, оставленных занятию, делятся на ядра и память посылки по умолчанию. Своё число владелец задаёт во вкладке «Ресурсы» панели, см. ресурсы. Очередь честная по людям: вторая посылка одного человека встаёт за первыми посылками остальных, поэтому один участник не займёт исполнитель на всю пару.
    Пульт соревнования (вкладка «Посылки») показывает, что исполняется сейчас, кто ждёт, пять чисел сводки и ленту посылок. В меню строки: открыть исполненную тетрадь, посмотреть весь вывод, исполнить заново, пересчитать метрику, не засчитывать. Полоса исполнителя в списке соревнований позволяет приостановить очередь, а у идущего прогона есть кнопка «Убить».
    Чтобы никто не подгонял решение под публичный лидерборд десятками посылок, поставьте дневную норму посылок. Норму тратят только посылки, получившие оценку. Упавшие — ошибка в тетради, лимит времени или памяти, непринятый ответ, сбой метрики или самого сервера — и отменённые её не тратят; посылка в очереди временно занимает место и возвращает его, если падает. От бесконечных падений защищает отдельный потолок упавших посылок: не больше 10 в день на участника или трёх норм, если это больше (сбои метрики в него не идут). Упёршийся в потолок участник видит свой отказ, отдельный от нормы. Без нормы (0) нет и потолка.
    
+
    Видеокарта для посылок
    Тетради, которая учит нейросеть, нужна видеокарта. Посылка получает долю карты, когда сошлись два условия: владелец сервера отдал посылкам карты, и соревнование карту просит.
    1. Владелец отдаёт карты. Во вкладке «Ресурсы» панели блок «Видеокарты для посылок» перечисляет карты машины (KERNEL_GPUS) с названиями и памятью. Отметьте те, что служат посылкам, и задайте, сколько посылок делят одну карту одновременно, от 1 до 8. Отданная посылкам карта уходит из пула комнат, поэтому занятие и посылка никогда не делят одну карту; занятие, которое держит карту в этот момент, сохраняет её, пока работает его контейнер, а посылки до тех пор её ждут; и наоборот, карта, возвращённая комнатам, начинает служить занятию, когда закончатся идущие на ней посылки. На машине, настроенной без панели, то же задают COMPETITION_GPUS (элементы KERNEL_GPUS через запятую) и COMPETITION_GPU_PER_DEVICE; сохранённое в панели главнее.
    2. Соревнование просит карту. В редакторе выберите окружение с CUDA (base-gpu, kaggle-gpu или своё поверх них) и включите переключатель «Видеокарта». Он есть только у таких окружений, и пока он включён, окружение нельзя сменить на окружение без CUDA. Карту получает только тетрадь. Метрика не получает никогда: у неё скрытые ответы, и учить ей нечего.
    Доля карты. Посылки на одной карте получают поровну её памяти: на карте 24 ГБ при трёх одновременно — по 8 ГБ. Если карты разного размера, обещанная доля — по наименьшей из них: её хватит на любой карте. Число посылок на карту можно менять и посреди занятия: идущая посылка сохраняет долю, с которой началась, а новую карта берёт, только когда её доля помещается рядом с их долями, — поэтому карта, которую одна посылка занимает целиком, после перехода с 1 на 2 никого больше не возьмёт, пока та не закончится.
    Как держат долю. С двух сторон. Внутри посылки PyTorch ограничен долей за вычетом 512 МБ на контекст CUDA, даже если какая-нибудь библиотека, например polars, ищет его раньше, чем тетрадь его импортирует: модель, которая выросла из ограничения, падает в своей ячейке с torch.OutOfMemoryError, как на карте поменьше. CatBoost на GPU по умолчанию сразу берёт 95 % свободной памяти карты; в доле его gpu_ram_part — та часть карты, которую составляет доля за вычетом контекста CUDA (около 0,48 при двух посылках на карте 24 ГБ), если тетрадь не задала свою. XGBoost на device="cuda" берёт память по ходу дела, а не заранее. TensorFlow и JAX в своём окружении не ограничиваются: по умолчанию оба сразу занимают большую часть карты, и исполнитель останавливает их за это, если тетрадь не придержит их до импорта (TF_FORCE_GPU_ALLOW_GROWTH=true, XLA_PYTHON_CLIENT_PREALLOCATE=false). Снаружи исполнитель каждые 3 секунды смотрит, сколько памяти карты на самом деле держит контейнер посылки, PyTorch это или нет, и останавливает посылку, которая дважды подряд превысила долю больше чем на 256 МБ. Смотрит он от root, которому код тетради не может послать сигнал. А посылку, карту которой 4 раза подряд не удалось прочитать (около 12 секунд), тоже останавливают: прогон, за которым никто не смотрит, может держать всю карту. Если на карте одна посылка, карта целиком её, и ничего не ограничивается и не проверяется. Если карта не сообщила свой объём, обещать нечего: панель пишет, что размер неизвестен, и ничего не ограничивается и не проверяется тоже.
    Что видит участник. На странице соревнования доля карты стоит рядом с остальными пределами. Тетрадь, упёршаяся в ограничение изнутри, — это «Упала тетрадь» с тем самым torch.OutOfMemoryError и нашей строкой о её доле; посылка, остановленная снаружи, — «Не хватило памяти» с её долей и тем, сколько она заняла. Со скрытым тестом при варианте «Только статус и тип ошибки», который стоит по умолчанию, в сообщении только доля: сколько заняла посылка — число, которое выбрала её тетрадь (см. скрытый тест). Посылка, карта которой перестала отвечать, останавливается как «Упала тетрадь» и считается упавшей: так бывает, когда тетрадь занимает процессор сотнями потоков или процессов или когда сама карта перестаёт отвечать. Если же карта не ответила за весь прогон ни разу, виноват хост, а не тетрадь: посылка заканчивается сбоем самого сервера, который ни во что не засчитывается, а в журнале преподавателя сказано проверить nvidia-smi внутри контейнера. Всплеск мимо ограничения может за секунды до второго замера отнять память у соседей по карте: посылку, которой не хватило видеопамяти (torch.OutOfMemoryError), пока исполнитель останавливал другую посылку на той же карте за выход за долю, один раз возвращают в очередь, с нашей строкой об этом, и эта попытка не засчитывается. Если посылкам не отдано ни одной карты, соревнование с «Видеокартой» сразу отказывает в загрузке и в проверке бейслайна и называет причину: никто не ждёт в очереди карту, которая не придёт.
    Чего доля не даёт. Посылки на одной карте пользуются её вычислениями по очереди: модель, которая держит карту занятой, при трёх посылках на карте учится примерно втрое медленнее, чем одна, а предел времени тетради считает это время. Доля бережёт память, а не скорость: поднимите предел времени или посадите на карту меньше посылок. Небольшая модель между своими шагами оставляет карту свободной и соседей почти не замечает, см. замер ниже. В очереди GPU-посылке нужны свободный исполнитель и свободная доля карты; пока она ждёт карту, процессорные посылки за ней проходят вперёд.
    Что карта даёт занятию. Замер 13 октября 2026 года на Vast VM с 61 vCPU, 98 ГБ памяти и 2 × RTX 4090 (по 24 ГБ): 25 участников в один момент отправили эталонное решение соревнования по распознаванию букв — небольшую свёрточную сеть; 2 ядра, 4 ГБ и 600 секунд на посылку; kaggle-gpu, — и каждая посылка получила оценку. Прогон — время, которое посылка исполнялась; вердикт считается от её загрузки.
    Карты для посылокПрогон, медианаВердикт, медианаВердикт, последний
    нет: только процессор, на 0.21.0 (22 исполнителя по 2 ядра)159 с162 с235 с
    1 карта × 425 с102 с175 с
    2 карты × 429 с58 с88 с
    1 карта × 8 (доля около 3 ГБ)29 с60 с117 с
    2 карты × 831 с36 с62 с
    Карта сокращает один прогон этой тетради в пять-шесть раз, и дальше занятие ждёт очередь, а не прогон: восемь посылок на карте шли почти так же быстро, как четыре, а две карты по восемь выдали вердикт всем 25 примерно за минуту. Тяжёлой модели лишние посылки на карте помогают меньше, и ей ещё нужно уместиться в долю поменьше. При смешанной нагрузке — 10 GPU-посылок и 4 процессорных на той же машине — очередь держала своё правило: процессорные посылки не ждали за GPU-посылками.
    На Kubernetes долю определяет кластер: значение чарта gpu.competitions.resource называет ресурс device plugin, который просит GPU-посылка, — целую карту, реплику time-slicing или часть MIG; см. эксплуатацию. Сколько GPU-посылок одновременно уходит в кластер, задаёт gpu.competitions.slots, по одной, пока оно не задано: очередь держит там столько и пропускает процессорные посылки мимо остальных, как мимо занятой карты здесь. Сам Colloq там ничего не ограничивает и не проверяет: у части MIG своя память, а реплики time-slicing делят память карты без всякой границы между собой. И карта там не уходит из пула комнат: GPU-комнаты просят nvidia.com/gpu, поэтому с этим именем посылки берут единицы с карт комнат, и GPU-комната, которой не хватило свободной карты, не запустится, пока их держат посылки. Если на занятиях работают GPU-комнаты, дайте посылкам собственный ресурс, см. узлы с GPU.
    
+
+
    Очередь и пульт
    Очередь одна на весь инстанс: она лежит в базе и переживает перезапуск сервера. Сколько посылок исполняется одновременно, по умолчанию считается по машине («Авто»): её ядра и память за вычетом 2 ядер и 2 ГБ, оставленных занятию, делятся на ядра и память посылки по умолчанию. Своё число владелец задаёт во вкладке «Ресурсы» панели, см. ресурсы. Очередь честная по людям: вторая посылка одного человека встаёт за первыми посылками остальных, поэтому один участник не займёт исполнитель на всю пару.
    Пульт соревнования (вкладка «Посылки») показывает, что исполняется сейчас, кто ждёт, пять чисел сводки и ленту посылок. В меню строки: открыть исполненную тетрадь, посмотреть весь вывод, исполнить заново, пересчитать метрику, не засчитывать. Полоса исполнителя в списке соревнований позволяет приостановить очередь, а у идущего прогона есть кнопка «Убить». Владелец удаляет соревнование пунктом «Удалить соревнование» в меню его строки в списке, и сначала уходят его прогоны: ждущие посылки покидают очередь, а идущие убиваются, и удаление ждёт их остановки, до 30 секунд на каждую. Если какая-то за это время не остановилась, ничего не удаляется, хотя ждавшие посылки остаются отменёнными, и удаление можно повторить через минуту.
    Чтобы никто не подгонял решение под публичный лидерборд десятками посылок, поставьте дневную норму посылок. Норму тратят только посылки, получившие оценку. Упавшие — ошибка в тетради, лимит времени или памяти, непринятый ответ, сбой метрики или самого сервера — и отменённые её не тратят; посылка в очереди временно занимает место и возвращает его, если падает. От бесконечных падений защищает отдельный потолок упавших посылок: не больше 10 в день на участника или трёх норм, если это больше (сбои метрики в него не идут). Упёршийся в потолок участник видит свой отказ, отдельный от нормы. Без нормы (0) нет и потолка.
    
 
 
    Участники и ключ входа
    Участник соревнования — не житель комнаты: его личность принадлежит всему инстансу. Вступая, человек указывает логин Telegram или почту — под этим именем он стоит в лидерборде — и получает ключ входа вида K7Q-M2X-9FD и ссылку к нему. Тем же ключом он возвращается к своим посылкам с другого устройства, из другого браузера или после очистки данных. Логин или почта — это один человек: под именем, которое в соревновании уже есть, вступить нельзя. Отказ предлагает войти по своему ключу, и второй личности с другим ключом не появляется.
    Ключ лежит в карточке «ВАШ КЛЮЧ ВХОДА» на странице соревнований — её видно каждый раз, а не однажды. В базе самого ключа нет: хранятся его отпечаток и копия, зашифрованная секретом инстанса, поэтому украденная база без секрета не открывает ни одного входа.
    Если участник ключ потерял, выдайте новый на вкладке «Участники». Старый перестаёт действовать в ту же секунду — вместе со всеми вкладками, которые были на нём открыты.
    
 
 
    Дедлайн и разбор
    Когда дедлайн проходит, приём посылок закрывается сам, а итоговый лидерборд открывается — если в настройках выбрано «открыть сам после дедлайна». Второй вариант — «Открою вручную — на разборе»: тогда итоги показывает кнопка, и вы решаете, в какую минуту пары их увидят.
    «Завершить сейчас» закрывает приём раньше срока. После дедлайна нажимать её не нужно: соревнование и так считается завершённым — в списке у участников и в панели оно стоит среди завершённых, а в шапке пульта вместо «Завершить сейчас» появляются «Закрыть поздний приём» или «Принимать поздние» и, если итоги открываются вручную, «Открыть приватный лидерборд».
    Тем, кто доделывает решение после звонка, поможет переключатель «Поздние посылки» в разделе «Сроки и зачёт». С ним после дедлайна и после «Завершить сейчас» посылки принимаются дальше: исполняются, получают публичный счёт и тратят дневной лимит по тем же правилам, что и обычные. Но в зачёт поздняя посылка не идёт никогда: её нет в местах, лидербордах, сводке и ожидании итогов, выбрать её для зачёта нельзя. Итоговый счёт участник увидит, когда откроются итоги. У него на странице она отмечена «после дедлайна · вне зачёта», а на пульте у неё отметка «ПОЗДНЯЯ» и свой фильтр «Поздние». Поздней становится посылка, загрузка которой началась после дедлайна; начатая вовремя и докачанная в пределах двух минут остаётся в зачёте. Ждущую проверки посылку, отправленную вовремя, поздняя не заменяет. Пока приём поздних открыт, это видно в списках: у участников соревнование стоит среди завершённых с отметкой «приём поздних посылок открыт» и кнопкой, а в панели под «ЗАВЕРШЕНО» написано «поздние принимаются» и сколько их пришло.
    Если на идущем соревновании перенести дедлайн позже, посылки, отправленные до нового дедлайна, становятся обычными: попадают в лидерборд и в зачёт. Перенос раньше никого поздним не делает. После «Завершить сейчас» перенос ничего не меняет: итоги закрыты вручную.
    На разборе пригодится «Лидерборд на проектор» — отдельное окно с одной таблицей, без имён файлов и служебных отметок.
    
 
-
    Дополнительные пакеты
    Во вкладке «Пакеты» участник видит закреплённую версию Python и полный состав базы. Дополнительные библиотеки можно указать по одной на строку или загрузить из requirements.txt. Комментарии — отдельной строкой или после запроса (numpy==1.26  # для метрики) — и пустые строки не входят в лимит 32 запросов; весь файл ограничен 8 КиБ. Сервер заранее проверяет совместимость, скачивает готовые wheels из PyPI и проверяет установку без сети. Версии базовых пакетов менять нельзя.
    Готовый набор выбирается перед явным подтверждением отправки тетради. Он и базовый образ закрепляются за посылкой; изменение выбора не меняет повтор старой посылки. Набор ставится в память посылки при каждом запуске и считается в её лимит: ему можно не больше половины, остальное — ядру, данным и рабочей папке. Набор больше этого не выбрать, и участник видит, сколько он займёт. На Docker-пути для подготовки требуется Docker Engine 28 или новее. Во время исполнения тетради и метрики интернет выключен.
    Проверки и повторное исполнение
    Каждая попытка получает отдельные каталоги и идентификатор. Пересчёт активной посылки ждёт окончания текущей попытки; старый CSV не может стать результатом неудачного нового запуска. Изменение метрики, данных, базового окружения или стартовой тетради требует актуальной проверки бейслайна. После изменения данных или окружения одного пересчёта метрики недостаточно: тетрадь нужно исполнить снова.
    Доступность исполнителя
    Посылки и подготовка пакетов исполняются там же, где ядра комнат: на Docker-пути (готовый образ сервера, colloq start) — в одноразовых контейнерах того же Docker, на k3s — в подах, которые запускает брокер. Если исполнитель или подготовка пакетов недоступны, интерфейс называет причину и не принимает работу в очередь; /api/health сообщает о каждой возможности отдельно.
    
-
+
    Дополнительные пакеты
    Во вкладке «Пакеты» участник видит закреплённую версию Python и полный состав базы. Дополнительные библиотеки можно указать по одной на строку или загрузить из requirements.txt. Комментарии — отдельной строкой или после запроса (numpy==1.26  # для метрики) — и пустые строки не входят в лимит 32 запросов; весь файл ограничен 8 КиБ. Сервер заранее проверяет совместимость, скачивает готовые wheels из PyPI и проверяет установку без сети. Версии базовых пакетов менять нельзя.
    Готовый набор выбирается перед явным подтверждением отправки тетради. Он и базовый образ закрепляются за посылкой; изменение выбора не меняет повтор старой посылки. Набор ставится в память посылки при каждом запуске и считается в её лимит: ему можно не больше половины, остальное — ядру, данным и рабочей папке. Набор больше этого не выбрать, и участник видит, сколько он займёт. На Docker-пути для подготовки требуется Docker Engine 28 или новее. Во время исполнения тетради и метрики интернет выключен.
    Проверки и повторное исполнение
    Каждая попытка получает отдельные каталоги и идентификатор. Пересчёт активной посылки ждёт окончания текущей попытки; старый CSV не может стать результатом неудачного нового запуска. Изменение метрики, данных, базового окружения или стартовой тетради требует актуальной проверки бейслайна. После изменения данных или окружения одного пересчёта метрики недостаточно: тетрадь нужно исполнить снова.
    Доступность исполнителя
    Посылки и подготовка пакетов исполняются там же, где ядра комнат: на Docker-пути (готовый образ сервера, colloq start) — в одноразовых контейнерах того же Docker, на k3s — в подах, которые запускает брокер. Если исполнитель или подготовка пакетов недоступны, интерфейс называет причину и не принимает работу в очередь; /api/health сообщает о каждой возможности отдельно. Соревнованию с «Видеокартой» нужна ещё и доля карты: если посылкам не отдано ни одной карты (на Kubernetes — не задан GPU-ресурс для посылок), оно отказывает в загрузке и в проверке бейслайна с этой причиной.
    
+
 
    Поиск по документации
    Начните вводить название или вопрос.
    Поиск работает в вашем браузереEsc закрыть
    
\ No newline at end of file
diff --git a/site/docs/en/competitions.html b/site/docs/en/competitions.html
index b914a850..1aa674a5 100644
--- a/site/docs/en/competitions.html
+++ b/site/docs/en/competitions.html
@@ -29,14 +29,16 @@ 
    The public and the private share
    The metric is computed tw
 
 
    The hidden test
    An open data/test.csv is downloaded by the class, and an answer is easy to fit to it: count the rows, copy the ids. A hidden test is a file the entrant does not download: at every check the notebook reads it at the same path, in place of the open example. The model is checked on rows nobody has seen.
    1. In the editor's Hidden test section, upload the file through its own button — not through the open data. In Path during the check give its name in data/; by default it is the file's own name. A file named like an open example replaces the example; a file with a new name is added to the data.
    2. The example stays with the entrants: they download it and debug their notebook on it. Check the columns — the editor shows whether they match the example. A notebook written for the example fails on a test with other columns.
    3. The answers in solution.csv must belong to the hidden test's rows.
    4. Run the sample notebook's check again: a change to the hidden test makes its last check stale.
    Every run gets a data/ directory of its own: the open files, with the hidden ones in their place. data/test.csv and /data/test.csv both read the hidden test; the metric never sees it. A file replaced in the middle of a run reaches the next runs only. The ceiling is 200 MB and 10 files, apart from the open data.
    Entrants see the hidden file's name and row count: the example carries an EXAMPLE mark in the data list, and the check conditions have a line about the hidden test. No link hands out its contents — not to the entrant and not to you: the hidden test cannot be downloaded.
    While the notebook runs, the hidden rows are in its memory, and any output can carry them out: test.head(), a KeyError message with a value in it, a metric message quoting the answer. That is why What participants see of a check has two options:
    • Only the outcome and the error type — the default. The entrant sees the outcome, the number of the failed cell and the exception type from a short list of common ones (KeyError, ValueError…). No cell output, no error text, no traceback; the notebook link hands out the notebook as it was sent, not the executed one. A metric refusal is retold in our own words, without values, ids or counts from the answer.
    • Full output — as without a hidden test. Turn it on only if revealing those rows does no harm: fitting the answer to them becomes possible.
    In the feed you always see the whole output and the executed notebook. The full text is kept under either option, so a switch applies to every submission at once, with nothing run again.
    
 
-
    How a submission is executed
    The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in.
    Entrants download the open files one by one, or with Download all above the list: the archive holds data/ with every file and the starter notebooks next to it, as in the check, so pd.read_csv('data/train.csv') works right after unpacking. Hidden answers and the hidden test are never in it.
    1. The submission joins the instance queue.
    2. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv.
    3. That file is taken out of the container and the container is destroyed.
    4. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here.
    The competition's limits (notebook time, memory, cores) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full. Each submission has a "Download the notebook with its output" link: the notebook as the server executed it — a failed one too, up to the cell where it stopped. Its output is this run's alone: whatever came from the laptop is cleared before the run.
    OutcomeWhat the entrant sees
    DoneThe public score and their place.
    Notebook failed"Failed on cell 5 of 14" and their traceback verbatim.
    Time limit"Did not fit into 10 minutes, cell 6 of 50".
    Out of memory"Took more than 4 GB, cell 2".
    Answer not accepted"Reached the end but left no submission.csv", "The answer is missing 3 rows, for example id=17" — or the text of your ParticipantVisibleError.
    Metric failed"The checking code failed", with no detail; you get the traceback and the submission waits to be rescored.
    That is a competition without a hidden test. With one, an entrant by default sees a failed submission briefly: the outcome, the cell number and the exception type, with no output and no traceback; see the hidden test.
    
+
    How a submission is executed
    The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in.
    Entrants download the open files one by one, or with Download all above the list: the archive holds data/ with every file and the starter notebooks next to it, as in the check, so pd.read_csv('data/train.csv') works right after unpacking. Hidden answers and the hidden test are never in it.
    1. The submission joins the instance queue.
    2. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv.
    3. That file is taken out of the container and the container is destroyed.
    4. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here.
    The competition's limits (notebook time, memory, cores, and the GPU switch) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full. Each submission has a "Download the notebook with its output" link: the notebook as the server executed it — a failed one too, up to the cell where it stopped. Its output is this run's alone: whatever came from the laptop is cleared before the run.
    OutcomeWhat the entrant sees
    DoneThe public score and their place.
    Notebook failed"Failed on cell 5 of 14" and their traceback verbatim.
    Time limit"Did not fit into 10 minutes, cell 6 of 50".
    Out of memory"Took more than 4 GB, cell 2".
    Answer not accepted"Reached the end but left no submission.csv", "The answer is missing 3 rows, for example id=17" — or the text of your ParticipantVisibleError.
    Metric failed"The checking code failed", with no detail; you get the traceback and the submission waits to be rescored.
    That is a competition without a hidden test. With one, an entrant by default sees a failed submission briefly: the outcome, the cell number and the exception type, with no output and no traceback; see the hidden test.
    
 
-
    The queue and the console
    There is one queue for the whole instance; it lives in the database and survives a server restart. How many submissions run at once is worked out from the machine by default (Auto): its cores and memory, less 2 cores and 2 GB kept for the class, divided by a new submission's default cores and memory. The owner can set a fixed number in the panel's Resources tab; see sizing. The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session.
    The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue, and a run in progress has a Kill button.
    To stop anyone from fitting the public leaderboard with dozens of submissions, set a daily submission quota. Only submissions that get a score spend it. Failed ones — a notebook error, the time or memory limit, a rejected answer, a failure of the metric or of the server itself — and cancelled ones do not; a queued submission holds a place for now and gives it back if it fails. A separate ceiling on failed submissions stops endless failures: at most 10 a day per entrant, or three times the quota if that is more (metric failures do not count toward it). An entrant who reaches it gets a refusal of its own, apart from the quota. With no quota (0) there is no ceiling either.
    
+
    A card for submissions
    A notebook that trains a network wants a video card. A submission gets a share of one when two things are true: the server owner has given cards to submissions, and the competition asks for one.
    1. The owner gives cards. In the panel's Resources tab, the GPUs for submissions block lists the machine's cards (KERNEL_GPUS) with their names and memory. Tick the ones that serve submissions and say how many submissions share one card at a time, from 1 to 8. A card given to submissions leaves the rooms' pool, so a class and a submission never share a card; a class that holds the card at that moment keeps it until its container ends, and submissions wait for it until then; the other way round, a card given back to the rooms serves a class once the submissions running on it end. On a machine set up without the panel, COMPETITION_GPUS (entries of KERNEL_GPUS, comma-separated) and COMPETITION_GPU_PER_DEVICE do the same; what the panel saved wins.
    2. The competition asks. In the editor, pick an environment with CUDA (base-gpu, kaggle-gpu or one of your own on top of them) and turn the GPU switch on. Only such environments have it, and while it is on the environment cannot be changed to one without CUDA. Only the notebook gets the card. The metric never does: it holds the hidden answers and has nothing to train.
    The share. Submissions on one card get equal parts of its memory: on a 24 GB card with 3 at a time, 8 GB each. With cards of different sizes, the share promised is the smallest card's: it holds on whichever card a run lands. The number per card may change during a class: a running submission keeps the share it started with, and a card takes a new one only once its share fits beside theirs, so a card raised from 1 to 2 while a submission has it whole takes nobody else until that one ends.
    How the share is held. From two sides. Inside the submission, PyTorch is capped at the share less 512 MB for the CUDA context, also when a library such as polars looks for it before the notebook imports it: a model that outgrows the cap fails in its cell with torch.OutOfMemoryError, as it would on a smaller card. CatBoost on the GPU takes 95 % of the card's free memory at once by default; within a share its gpu_ram_part becomes the share's part of the card less the CUDA context (about 0.48 with 2 to a 24 GB card), unless the notebook sets its own. XGBoost on device="cuda" takes memory as it goes, not up front. TensorFlow and JAX, in an environment of your own, are not capped: by default both take most of the card at once, and the runner stops them for it, unless the notebook holds them back before importing them (TF_FORCE_GPU_ALLOW_GROWTH=true, XLA_PYTHON_CLIENT_PREALLOCATE=false). From outside, every 3 seconds the runner reads how much of the card the submission's container really holds, PyTorch or not, and stops a submission that is more than 256 MB over its share twice in a row. It reads as root, which the notebook's own code cannot signal, and it fails closed: a submission whose card cannot be read 4 times in a row (about 12 seconds) is stopped too, because a run nobody watches may be holding the whole card. With 1 submission per card, the whole card is the submission's, and nothing is capped or watched. A card that cannot tell its memory has no share to promise: the panel says its size is unknown, and nothing is capped or watched either.
    What the entrant sees. The competition page shows the share of the card next to the other limits. A notebook that hit the cap inside ends as Notebook failed with that torch.OutOfMemoryError and a line of ours naming its share; a submission stopped from outside ends as Out of memory, naming its share and how much it took. With a hidden test, under the default Only the outcome and the error type, the message names only the share: how far past it a run went is a number its notebook chose (see the hidden test). A submission whose card stopped answering is stopped as Notebook failed and counts as a failed one: that happens when a notebook keeps its processor busy with hundreds of threads or processes, or when the card itself stops answering. If the card never answered during the run, the host is to blame, not the notebook: the submission ends as the server's own failure, which counts toward nothing, and the teacher's log says to check nvidia-smi inside a container. A burst outside the cap can take a neighbour's memory in the seconds before the runner's second reading: a submission that runs out of video memory (torch.OutOfMemoryError) while the runner is stopping another one on the same card for going past its share goes back to the queue once, with a line saying so, and that attempt does not count. If no card is given to submissions, a competition with GPU on refuses uploads and the baseline check outright and says why: nothing waits in the queue for a card that will not come.
    What a share does not give. Submissions on one card take turns at its compute: three at a time, a model that keeps the card busy trains about three times slower than alone, and the notebook time limit counts that time. The share protects memory, not speed: raise the time limit, or put fewer submissions on a card. A small model leaves the card idle between its steps and hardly notices its neighbours; see the measurement below. In the queue a GPU submission needs a free runner and a free share of a card; while it waits for the card, CPU submissions behind it go ahead.
    What a card gives a class. Measured on 13 October 2026 on a Vast VM with 61 vCPU, 98 GB of memory and 2 × RTX 4090 (24 GB each): 25 entrants sent the reference solution of a letter-recognition competition — a small convolutional network; 2 cores, 4 GB and 600 seconds per submission; kaggle-gpu — at the same moment, and every submission scored. A run is the time a submission spent running; a verdict counts from its upload.
    Cards for submissionsRun, medianVerdict, medianVerdict, last
    none: CPU only, on 0.21.0 (22 runners × 2 cores)159 s162 s235 s
    1 card × 425 s102 s175 s
    2 cards × 429 s58 s88 s
    1 card × 8 (a share of about 3 GB)29 s60 s117 s
    2 cards × 831 s36 s62 s
    A card makes one run of this notebook five to six times shorter, and after that the class waits for the queue, not for the run: eight to a card ran nearly as fast as four, and two cards with eight each gave all 25 their verdict in about a minute. A bigger model gains less from more submissions to a card, and must fit into the smaller share. A mixed load of 10 GPU and 4 CPU submissions on the same machine kept the queue's rule: the CPU ones did not wait behind the GPU ones.
    On Kubernetes the cluster decides what a share is: the chart value gpu.competitions.resource names the device-plugin resource a GPU submission requests, a whole card, a time-sliced replica or a MIG slice; see operations. How many GPU submissions go to the cluster at once is gpu.competitions.slots, one while it is unset: the queue keeps that many there and lets CPU submissions go past the rest, as past a full card here. Colloq caps and watches nothing there itself: a MIG slice has memory of its own, while time-sliced replicas share the card's memory with no limit between them. Nor does a card leave the rooms there: GPU rooms request nvidia.com/gpu, so with that name submissions take their units from the rooms' cards, and a GPU room that finds none free is refused while submissions hold them. Where classes use GPU rooms, give submissions a resource of their own; see GPU nodes.
    
+
+
    The queue and the console
    There is one queue for the whole instance; it lives in the database and survives a server restart. How many submissions run at once is worked out from the machine by default (Auto): its cores and memory, less 2 cores and 2 GB kept for the class, divided by a new submission's default cores and memory. The owner can set a fixed number in the panel's Resources tab; see sizing. The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session.
    The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue, and a run in progress has a Kill button. An owner deletes a competition with Delete competition in its row's menu in the list, and its runs go first: the waiting submissions leave the queue, and the running ones are killed and waited for, up to 30 seconds each. If one has not stopped by then, nothing is deleted, though the waiting ones stay cancelled, and the delete can be repeated a minute later.
    To stop anyone from fitting the public leaderboard with dozens of submissions, set a daily submission quota. Only submissions that get a score spend it. Failed ones — a notebook error, the time or memory limit, a rejected answer, a failure of the metric or of the server itself — and cancelled ones do not; a queued submission holds a place for now and gives it back if it fails. A separate ceiling on failed submissions stops endless failures: at most 10 a day per entrant, or three times the quota if that is more (metric failures do not count toward it). An entrant who reaches it gets a refusal of its own, apart from the quota. With no quota (0) there is no ceiling either.
    
 
 
    Entrants and the sign-in key
    An entrant is not a room member: their identity belongs to the whole instance. On joining, a person gives their Telegram username or email — the name they appear under on the leaderboard — and receives a sign-in key shaped K7Q-M2X-9FD and a link carrying it. The same key brings them back to their submissions from another device, another browser, or after clearing site data. A username or an address is one person: joining under one the competition already has is refused with a pointer to the sign-in key, and no second identity with another key is created.
    The key sits in the "YOUR SIGN-IN KEY" card on the competitions page — shown every time, not once. The key itself is not in the database: what is stored is its fingerprint and a copy encrypted with the instance secret, so a stolen database without that secret opens no one's account.
    If an entrant loses their key, issue a new one from the Participants tab. The old one stops working that same second, together with every tab still open on it.
    
 
 
    The deadline and the debrief
    When the deadline passes, submissions close by themselves and the final leaderboard opens — if the settings say "open it by itself after the deadline". The other option is I will open it by hand — at the review: then a button reveals the results and you choose the minute of the class when they appear.
    Finish now closes submissions early. After the deadline there is no need to press it: the competition counts as finished anyway — participants and the panel list it among the finished ones, and the console header offers Close late intake or Take late submissions instead of Finish now, plus Open the private leaderboard when the results open by hand.
    For those finishing their solution after the bell there is the Late submissions switch in Dates and standings. With it on, submissions keep being accepted after the deadline and after Finish now: they run, get a public score and spend the daily quota by the same rules as ordinary ones. But a late submission never counts: it is in no places, no leaderboard, no summary and not in the wait for the final results, and it cannot be picked to count. The entrant sees its final score when the results open. On their page it is marked “after the deadline · not counted”, and in the console it carries a LATE mark and a Late filter of its own. A submission is late when its upload began after the deadline; one begun on time and finished within two minutes stays in the standings. A late submission never takes the place of an on-time one still waiting for its check. While late intake is open, the lists say so: participants see the competition among the finished ones marked “late submissions open”, with a button, and the panel writes “late ones accepted” under FINISHED, with how many came.
    If you move the deadline later on a running competition, submissions sent before the new deadline become ordinary ones: they join the leaderboard and the standings. Moving it earlier makes no one late. After Finish now moving it changes nothing: the standings were closed by hand.
    For the debrief there is Leaderboard on the projector — a separate window with just the table, without file names or internal marks.
    
 
-
    Additional packages
    The Packages tab shows the pinned Python version and complete base inventory. Entrants can enter additional libraries one per line or import requirements.txt. Comments, on a line of their own or after a requirement (numpy==1.26  # for the metric), and blank lines do not count towards the 32-requirement limit; the whole request is limited to 8 KiB. The server checks compatibility, downloads PyPI wheels and verifies installation without network access. Base package versions cannot be replaced.
    Choose a ready set before explicitly confirming notebook submission. Its packages and base image remain bound to that submission; changing the current selection does not alter a rerun. The set is installed into the submission's memory at every run and counts against its limit: it may take at most half, the rest being the kernel's, the data's and the working folder's. A bigger set cannot be chosen, and the entrant sees how much it would take. On the Docker path, preparation requires Docker Engine 28 or newer. Notebook execution and scoring have no internet access.
    Checks and reruns
    Every attempt has its own identifier and directories. Rescoring an active submission waits for that attempt to finish; an old CSV cannot become the result of a failed new run. Changes to the metric, data, base environment or starter notebook require an up-to-date baseline check. After changing notebook data or its environment, metric-only rescoring is insufficient: rerun the notebook.
    Executor availability
    Submissions and package preparation run where the rooms' kernels run: on the Docker path (the server image, colloq start) in throwaway containers on the same Docker, on k3s in Pods the broker starts. If the executor or package preparation is unavailable, the interface says why and refuses the work before it enters the queue; /api/health reports each capability separately.
    
-
+
    Additional packages
    The Packages tab shows the pinned Python version and complete base inventory. Entrants can enter additional libraries one per line or import requirements.txt. Comments, on a line of their own or after a requirement (numpy==1.26  # for the metric), and blank lines do not count towards the 32-requirement limit; the whole request is limited to 8 KiB. The server checks compatibility, downloads PyPI wheels and verifies installation without network access. Base package versions cannot be replaced.
    Choose a ready set before explicitly confirming notebook submission. Its packages and base image remain bound to that submission; changing the current selection does not alter a rerun. The set is installed into the submission's memory at every run and counts against its limit: it may take at most half, the rest being the kernel's, the data's and the working folder's. A bigger set cannot be chosen, and the entrant sees how much it would take. On the Docker path, preparation requires Docker Engine 28 or newer. Notebook execution and scoring have no internet access.
    Checks and reruns
    Every attempt has its own identifier and directories. Rescoring an active submission waits for that attempt to finish; an old CSV cannot become the result of a failed new run. Changes to the metric, data, base environment or starter notebook require an up-to-date baseline check. After changing notebook data or its environment, metric-only rescoring is insufficient: rerun the notebook.
    Executor availability
    Submissions and package preparation run where the rooms' kernels run: on the Docker path (the server image, colloq start) in throwaway containers on the same Docker, on k3s in Pods the broker starts. If the executor or package preparation is unavailable, the interface says why and refuses the work before it enters the queue; /api/health reports each capability separately. A competition with GPU on also needs a share of a card: with no card given to submissions (on Kubernetes, no GPU resource set for them) it refuses uploads and the baseline check with that reason.
    
+
 
    Search the docs
    Start typing a title or a question.
    Search runs in your browserEsc close
    
\ No newline at end of file
diff --git a/site/docs/en/environments.html b/site/docs/en/environments.html
index 1f42d9d7..0e3b1162 100644
--- a/site/docs/en/environments.html
+++ b/site/docs/en/environments.html
@@ -10,7 +10,7 @@ 
    Building a release (maintainers)
    The Publi
   --k3s-version EXACT_TESTED_K3S_VERSION --environments base,kaggle-base
    For GPU, also pass --gpu-toolkit-version EXACT_NVIDIA_APT_VERSION and --gpu-device-plugin-image NVIDIA_PLUGIN_IMAGE_AT_SHA256. Log in to the registry on the build machine. The k3s version must be v1.33 or newer: a room's memory is changed during class through the pods/resize subresource, and manifest validation rejects an older one. Publishing the artifacts is not, by itself, a test on the target VM.
    
 
    Memory and CPU for a class
    The Resources section is in the new class form and in the settings of an existing class (class menu → Settings). The Memory for the room's kernel, GB field moves in 0.5 GB steps, from 0.5 GB up to the machine's memory minus 1 GB. The Processor, cores field takes a whole number from 1 up to the machine's core count. To go back to the shared value, clear the field or click back to the environment default or back to the instance default.
    The hint under the field shows the total memory, the free memory and the selected environment's default. If Docker runs in its own virtual machine, as Docker Desktop and Colima do on macOS, the hint starts with Rooms live in Docker. In that case the memory size, the core count and the fields' upper limits come from Docker, not from the computer. Free memory is Docker's memory minus 1 GB and minus the limits of rooms that are already running. If free memory cannot be determined, the hint leaves it out. The form does not stop you from setting a limit above free memory, but it warns that the kernel may not start. For a running room, only the increase over what it already has is compared with free memory. If the machine has an NVIDIA graphics card, the section shows it and whether the selected environment uses the card. GPU memory is not limited: all rooms on the card share it.
    In Docker, a new memory limit applies to the running kernel immediately, without a restart; going back to the environment default takes effect the next time the room's container starts. A new core count also applies to the running kernel immediately, without a restart, and so does going back to the instance default. The numpy and torch thread count (OMP_NUM_THREADS and related variables) is set when the room's container is created and does not change for a running container, not even with a kernel restart: the restarted kernel inherits the same container's environment. The next container of the room gets the new thread count, for example when the room has stood empty for two hours and is opened again. Until then you can raise the thread count right in the notebook: threadpoolctl.threadpool_limits(n) for numpy and scikit-learn, torch.set_num_threads(n) for torch. Without KERNEL_MEM, the default is 4 GB, or 16 GB for a GPU environment. KERNEL_MEM sets the default for ordinary environments (a GPU environment stays at 16 GB), KERNEL_MEM_<ENVIRONMENT> sets it for one environment (for example, KERNEL_MEM_BASE_GPU=16g), and KERNEL_CPUS sets the number of cores (2 by default). The .env file that colloq start creates already contains KERNEL_MEM=4g. The owner can change these instance defaults without a restart (/api/admin/resources): a saved value wins over .env until it is cleared. The class's second container — the one where students' personal notebooks run — gets the same memory and CPU numbers, in a cgroup of its own, so a greedy draft does not take memory away from the lecture. It is never given a GPU. KERNEL_OWN_MAX (60 by default) sets how many kernels may live in it at once: over that, a run in a personal notebook is refused in words. Its process ceiling is its own, KERNEL_OWN_PIDS (2048 by default) — it holds dozens of kernels, and the room's 512 would run out at about thirty. A personal notebook's kernel that has computed nothing for KERNEL_OWN_IDLE_MIN minutes (30 by default; 0 never stops them) is stopped: the notebook is told in the kernel log, the next run brings it back, and the container goes with the last kernel in it.
    A class's running container holds its whole memory limit even when it is nearly empty: while it runs, no other class can have that memory. A class with nobody inside and nothing computing lets its kernels go after KERNEL_ROOM_IDLE_MIN minutes (120 by default; 0 keeps them); the panel's Resources tab → «Default room» changes it too. The same tab's Running now section shows which classes hold memory, how much of it is really used and who is inside; the server owner can stop any class's kernels or all idle ones at once. A stopped kernel loses its variables, the files stay, and the next run brings it back. How this works in a k3s installation is described below.
    
 
    Room resources
    In a k3s installation, a room Pod gets 2Gi of memory, 2 CPUs and 2Gi of ephemeral storage by default; requests equal limits. The memory and the CPU count from the class's “Resources” section reach the Pod, and you can raise or lower them during class: the broker changes them on the running Pod in place, so the kernel does not restart and variables stay. The back to the environment default and back to the instance default buttons also apply at once. The Pod takes the numpy and torch thread count from its CPU limit when it starts and, as in Docker, does not change it for the running kernel, neither after an in-place change nor after a kernel restart; the room's next Pod gets the new count. The broker has one default for all environments, GPU ones included, and that is the default the form's hint shows. The field's upper limit is the node's memory minus 1 GB. The broker variables RUNTIME_KERNEL_MEMORY and RUNTIME_KERNEL_MEMORY_MAX set the default and the ceiling: put them in the same instance.env as the app settings (see installation configuration), for example RUNTIME_KERNEL_MEMORY=4Gi. Values are whole Mi or Gi from 64Mi to 256Gi, and the ceiling can't be below the default; the installer checks them before stopping rooms and keeps them across updates. Changing memory in place needs Kubernetes 1.33 or newer, and a release with an older k3s fails validation. These are resource limits, not a promise of a certain number of students or a certain model training speed.
    A room's memory and cores are reserved in full, even if Python never touches them. That is why the hint counts as free the node's memory minus 1 GB and minus the memory already given to running rooms. If the node can't cover an increase in memory or cores right now, the new value is saved and waits: Kubernetes applies it on its own once the resource frees up, for example when a neighbouring room stops. The broker does not set more than the node can ever provide: the running Pod keeps its previous value. A new room has nowhere to wait: if the node can't cover its memory, CPU or GPU, startup stops after 20 seconds. The teacher sees in the room how much it asked for and what to lower or close; students see a short note that Python can't start right now and that the teacher can see why. If code runs out of memory, k3s ends the room's whole Pod, not just the kernel as in Docker; the next run starts a new Pod, already with the new memory value.
    Memory in /dev/shm counts toward the Pod limit: 64 MiB by default for a CPU environment and 1 GiB for a GPU one. The installer sets a limit of 256 PIDs per Pod. A namespace ResourceQuota and any extra admission rules are up to the operator; no ResourceQuota ships with the installation. There is no enforced per-room file quota for the workspace; keep an eye on the separate file partition and on free space.
    
-
    GPU environments
    Each GPU room requests one nvidia.com/gpu and RuntimeClass nvidia. Device sharing is not turned on automatically. Several GPU rooms running at once need a matching number of available devices, or a sharing policy that has been separately designed and tested.
    The installer keeps the host driver, installs pinned versions of the NVIDIA Container Toolkit and device plugin, and runs a real PyTorch CUDA operation as a check. The check needs a free GPU; a card that is in use is not taken away from a running class.
    sudo scripts/cluster.sh gpu-preflight
+
    GPU environments
    Two GPU environments ship with Colloq: base-gpu, which is PyTorch and torchvision as CUDA wheels and nothing else, and kaggle-gpu on top of it, which is the kaggle-base set at the same pinned versions plus seaborn, OpenCV and timm. Neither image is published. On a machine of your own they are built where the card is: on first boot with KERNEL_PRELOAD=base,base-gpu,kaggle-gpu, or with Build in the Environments panel. base-gpu takes about 9.9 GB of disk, and kaggle-gpu about 2.8 GB more, 12.7 GB in all. Both carry torch 2.14.1 and torchvision 0.29.1, wheels built for CUDA 13.0, so the machine with the card needs an NVIDIA driver of 580 or newer: on an older one PyTorch imports but never sees the card (torch.cuda.is_available() is false). nvidia-smi shows the driver's version and the highest CUDA it supports in its header. For a release, name them in --environments together with the two GPU options above. A competition on either runs on the CPU until its GPU switch is on; see a card for submissions.
    Each GPU room requests one nvidia.com/gpu and RuntimeClass nvidia. Device sharing is not turned on automatically. Several GPU rooms running at once need a matching number of available devices, or a sharing policy that has been separately designed and tested.
    The installer keeps the host driver, installs pinned versions of the NVIDIA Container Toolkit and device plugin, and runs a real PyTorch CUDA operation as a check. The check needs a free GPU; a card that is in use is not taken away from a running class.
    sudo scripts/cluster.sh gpu-preflight
 sudo k3s kubectl get nodes \
   -o "custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
    An application that responds normally does not prove the GPU is ready. If the card is not allocated or CUDA does not work, find the cause before the class starts.
    
 
    Keep old images
    Historical catalog revisions are kept as long as rooms refer to them. Do not delete the corresponding images from the registry. An application backup contains the catalog and the digests, but not the image layers.
    
diff --git a/site/docs/en/install.html b/site/docs/en/install.html
index e80f2895..7c0f4388 100644
--- a/site/docs/en/install.html
+++ b/site/docs/en/install.html
@@ -4,7 +4,7 @@
 
    
 
    Your own serverCOLLOQ / GUIDE
    Installing on k3s
    A preview for those without a cluster: a script installs single-node k3s on Linux amd64, an exact release and a separate Pod for each room.
    
 
-
    What you need
    A k3s installation is designed for one Linux amd64 VM with systemd and IPv4. You need root access, curl, Python 3, iptables (plus ip6tables on a host with IPv6) and the standard Linux utilities. A GPU release needs Debian or Ubuntu with apt, the gnupg package and a working NVIDIA driver installed in advance: the installer does not replace it. You do not need to install Node.js or the Docker daemon on the server: the application and the kernels run on k3s/containerd.
    Set aside resources for k3s, the application and every room that runs at the same time. The number of connected students does not determine the load by itself: what matters is the code, memory, files and the number of active kernels. See resource planning.
    
+
    What you need
    A k3s installation is designed for one Linux amd64 VM with systemd and IPv4. You need root access, curl, Python 3, iptables (plus ip6tables on a host with IPv6) and the standard Linux utilities. A GPU release needs Debian or Ubuntu with apt, the gnupg package and a working NVIDIA driver of 580 or newer installed in advance: the installer does not replace it, and on an older driver the GPU environments' torch 2.14.1, built for CUDA 13.0, never sees the card. You do not need to install Node.js or the Docker daemon on the server: the application and the kernels run on k3s/containerd.
    Set aside resources for k3s, the application and every room that runs at the same time. The number of connected students does not determine the load by itself: what matters is the code, memory, files and the number of active kernels. See resource planning.
    
 
    Verify the release kit
    Get the artifacts through a trusted channel. Verify the checksums, unpack the tools archive and work from its directory. The manifest records the source commit, the real digests of the application, broker and environment images, the exact k3s version and the hashes of the tools.
    sha256sum -c SHA256SUMS
 tar -xzf colloq-deploy.tar.gz
 python3 scripts/release.py validate --release release.json
diff --git a/site/docs/en/kubernetes.html b/site/docs/en/kubernetes.html
index 2212c1fa..9194c994 100644
--- a/site/docs/en/kubernetes.html
+++ b/site/docs/en/kubernetes.html
@@ -5,9 +5,9 @@
 
    Your own serverCOLLOQ / GUIDE
    Colloq in your Kubernetes
    A Helm chart in one namespace of a corporate cluster: permissions, security, network, storage, a registry mirror, installing with Helm or ArgoCD, and operations.
    
 
    In short for the platform team
    What it isColloq is a collaborative Jupyter classroom: a web app, a runtime broker and a separate Pod with a Python kernel for each room. It goes into your cluster as a Helm chart, oci://ghcr.io/colloq-edu/charts/colloq, in one namespace. Open source, MIT license.
    What runsThe app Deployment, one replica: the database is SQLite. The broker Deployment, one replica: the only thing that talks to the Kubernetes API. A Pod and a Service for each running room. A second Pod and Service per room for students' personal notebooks. Short-lived competition Pods: submissions, the metric, package preparation. Room and competition Pods are created by the broker, not by the chart.
    What it needsOne namespace with namespace-admin rights; the broker's Role from Permissions; a StorageClass — ReadWriteOnce required, ReadWriteMany preferred; an ingress controller and a certificate; a registry mirror with the release's images.
    What it never needscluster-admin or cluster-scoped objects: Namespace, PersistentVolume, ClusterRole, CRD, DaemonSet, RuntimeClass. Privileged containers, hostPath, hostNetwork or hostPID. Root: every container runs as uid 1000. Internet access from the cluster.
    NetworkInbound: only the ingress controller to the app, port 3000. Inside the namespace: app → broker, app and broker → rooms. Outbound: broker → Kubernetes API, Pods → cluster DNS; optionally app → the AI model and an outbound proxy. Students' code gets DNS only; competition submissions get nothing.
    DataTwo PVCs: colloq-data — the SQLite database, keys, class page files, competition data and submissions, database snapshots; colloq-workspace — room files.
    BackupsVolume snapshots with your own tools (Velero, CSI), plus consistent database snapshots the app writes into the colloq-data volume itself once a day.
    
 
    Students' and teachers' browsersThe cluster's ingress controller, TLSApp · Deployment colloq-app, 1 replicaBroker · Deployment colloq-runtime, 1 replica
    Room A · Pod and ServiceRoom B · Pod and ServiceThe room's personal notebooksCompetition submission
    The app — a Node.js server with the built web interface and an SQLite database — keeps everything on two volumes and never calls the Kubernetes API. When someone joins a room or runs a cell, the app asks the broker for the room's Pod. The broker creates the Pod and a ClusterIP Service from a template fixed in its code, waits for Jupyter and hands the app its address and token; from then on the app talks to the room's Jupyter directly, on port 8888. A room's Pod lives while anyone is in the room and for 2 hours after everyone has left. Room Pods survive restarts and upgrades of the app and the broker: the new broker finds them by their labels, and Python variables stay.
    
-
    What we assume about your cluster
    This is a checklist for your team: go through it before installing. The answer to each item is either "yes, that's us" or a chart value to set.
    WhatWe assumeIf yours differs
    Kubernetes version1.30 or newer, upstream-compatible: vanilla, Deckhouse and the like. Changing a running room's memory and cores without a restart needs 1.33 or newer and the pods/resize permission.Older than 1.33, or no such permission: runtime.inPlaceResize: false. The rule is dropped from the Role, resizing a live room answers "unsupported" rather than failing, and the new number goes to the room's next Pod.
    NamespaceWe get one namespace with namespace-admin rights, and it holds one Colloq installation: object names are fixed (colloq-app, colloq-runtime, …). The chart creates no cluster-scoped objects.Your team creates the namespace, its labels and its quota. The chart creates the Role and RoleBinding itself, and Kubernetes won't let anyone grant a verb they don't hold: whoever installs the chart needs every verb in Permissions.
    Pod Security and policiesPod Security Admission restricted is enforced, and Kyverno or Gatekeeper require requests and limits on every container, probes, runAsNonRoot and readOnlyRootFilesystem, forbid privileged, hostPath and hostNetwork, admit images only from the internal registry and only by digest, and require the standard app.kubernetes.io/* labels.Every Colloq Pod passes restricted; details and exceptions are under Security. commonLabels adds your labels to every chart object and every broker Pod; rooms.podLabels and rooms.podAnnotations add them to broker Pods only.
    Network policiesNetworkPolicy is enforced (Calico, Cilium), and the namespace may be default-deny.The chart declares every flow it needs, ingress and egress; they are listed under Network. Say where the ingress controller lives (networkPolicy.ingressController.from), where cluster DNS lives (networkPolicy.dns.to), and the API servers' addresses (networkPolicy.kubeApiServer).
    Ingress and TLSIngress through ingress-nginx, with a configurable class and annotations. TLS: the organisation's certificate from an existing Secret or from cert-manager. Users come from the corporate network, over VPN or, when students are external, from the internet.Another controller: ingress.className and ingress.annotations with the same meaning — request bodies of 256 MB or more, timeouts of an hour or more, no response buffering. Or ingress.enabled: false: publish the Service colloq-app:3000 with your own object and set config.publicUrl.
    Sign-inStudents join with the class link, teachers sign in with a personal link. A proxy with SSO that forwards a signed JWT — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — may stand in front of Colloq.config.sso.*: teachers sign in with their corporate account, students who have one join under their own name; see Sign-in through your SSO. Without such a proxy, nothing to set.
    StorageA StorageClass with ReadWriteOnce always exists; one with ReadWriteMany (CephFS) may.By default the chart assumes ReadWriteOnce: everything that mounts Colloq's volumes runs on one node. With ReadWriteMany, set persistence.accessMode: ReadWriteMany and room Pods spread over the cluster; see Storage.
    No internetThe cluster has no internet access. Images come through a registry mirror (Harbor, Nexus), PyPI packages through a Nexus mirror, and the Oracle through an internal model with an OpenAI-compatible API, or it stays off. There may be an outbound HTTP proxy and an internal CA.global.imageRegistry — the mirror for every image, kernel images included; dependencies.indexUrl and dependencies.mirrorEgress — the PyPI mirror; config.ai.baseUrl — the model; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. See Images and the registry mirror.
    Secrets and GitOpsSecrets come from Vault through External Secrets, and deployment goes through ArgoCD.Every secret can come from an existing Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. With those set, the chart renders no random value, so ArgoCD sees no perpetual drift.
    Monitoring and logsPrometheus Operator; logs are collected from stdout and stderr.metrics.enabled turns on /metrics with a token, metrics.serviceMonitor.enabled adds a ServiceMonitor; without the Prometheus Operator, scrape /metrics your own way. config.logFormat: json writes one JSON object per line.
    BackupsVolume snapshots: Velero, CSI.The app itself writes consistent database snapshots into the colloq-data volume (config.dbSnapshotHours, config.dbSnapshotKeep), so a volume snapshot always carries a whole copy of the database.
    GPUGPU nodes are optional.GPU rooms take a RuntimeClass, a nodeSelector and tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations; see Operations.
    
+
    What we assume about your cluster
    This is a checklist for your team: go through it before installing. The answer to each item is either "yes, that's us" or a chart value to set.
    WhatWe assumeIf yours differs
    Kubernetes version1.30 or newer, upstream-compatible: vanilla, Deckhouse and the like. Changing a running room's memory and cores without a restart needs 1.33 or newer and the pods/resize permission.Older than 1.33, or no such permission: runtime.inPlaceResize: false. The rule is dropped from the Role, resizing a live room answers "unsupported" rather than failing, and the new number goes to the room's next Pod.
    NamespaceWe get one namespace with namespace-admin rights, and it holds one Colloq installation: object names are fixed (colloq-app, colloq-runtime, …). The chart creates no cluster-scoped objects.Your team creates the namespace, its labels and its quota. The chart creates the Role and RoleBinding itself, and Kubernetes won't let anyone grant a verb they don't hold: whoever installs the chart needs every verb in Permissions.
    Pod Security and policiesPod Security Admission restricted is enforced, and Kyverno or Gatekeeper require requests and limits on every container, probes, runAsNonRoot and readOnlyRootFilesystem, forbid privileged, hostPath and hostNetwork, admit images only from the internal registry and only by digest, and require the standard app.kubernetes.io/* labels.Every Colloq Pod passes restricted; details and exceptions are under Security. commonLabels adds your labels to every chart object and every broker Pod; rooms.podLabels and rooms.podAnnotations add them to broker Pods only.
    Network policiesNetworkPolicy is enforced (Calico, Cilium), and the namespace may be default-deny.The chart declares every flow it needs, ingress and egress; they are listed under Network. Say where the ingress controller lives (networkPolicy.ingressController.from), where cluster DNS lives (networkPolicy.dns.to), and the API servers' addresses (networkPolicy.kubeApiServer).
    Ingress and TLSIngress through ingress-nginx, with a configurable class and annotations. TLS: the organisation's certificate from an existing Secret or from cert-manager. Users come from the corporate network, over VPN or, when students are external, from the internet.Another controller: ingress.className and ingress.annotations with the same meaning — request bodies of 256 MB or more, timeouts of an hour or more, no response buffering. Or ingress.enabled: false: publish the Service colloq-app:3000 with your own object and set config.publicUrl.
    Sign-inStudents join with the class link, teachers sign in with a personal link. A proxy with SSO that forwards a signed JWT — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — may stand in front of Colloq.config.sso.*: teachers sign in with their corporate account, students who have one join under their own name; see Sign-in through your SSO. Without such a proxy, nothing to set.
    StorageA StorageClass with ReadWriteOnce always exists; one with ReadWriteMany (CephFS) may.By default the chart assumes ReadWriteOnce: everything that mounts Colloq's volumes runs on one node. With ReadWriteMany, set persistence.accessMode: ReadWriteMany and room Pods spread over the cluster; see Storage.
    No internetThe cluster has no internet access. Images come through a registry mirror (Harbor, Nexus), PyPI packages through a Nexus mirror, and the Oracle through an internal model with an OpenAI-compatible API, or it stays off. There may be an outbound HTTP proxy and an internal CA.global.imageRegistry — the mirror for every image, kernel images included; dependencies.indexUrl and dependencies.mirrorEgress — the PyPI mirror; config.ai.baseUrl — the model; outbound.httpsProxy, outbound.noProxy, outbound.extraCa. See Images and the registry mirror.
    Secrets and GitOpsSecrets come from Vault through External Secrets, and deployment goes through ArgoCD.Every secret can come from an existing Secret: config.existingSecret, secrets.runtimeToken.existingSecret, secrets.roomSecret.existingSecret, metrics.existingSecret. With those set, the chart renders no random value, so ArgoCD sees no perpetual drift.
    Monitoring and logsPrometheus Operator; logs are collected from stdout and stderr.metrics.enabled turns on /metrics with a token, metrics.serviceMonitor.enabled adds a ServiceMonitor; without the Prometheus Operator, scrape /metrics your own way. config.logFormat: json writes one JSON object per line.
    BackupsVolume snapshots: Velero, CSI.The app itself writes consistent database snapshots into the colloq-data volume (config.dbSnapshotHours, config.dbSnapshotKeep), so a volume snapshot always carries a whole copy of the database.
    GPUGPU nodes are optional.GPU rooms take a RuntimeClass, a nodeSelector and tolerations: gpu.runtimeClassName, gpu.nodeSelector, gpu.tolerations. Competition submissions get a GPU only through gpu.competitions.resource, the device-plugin resource they request; see Operations.
    
 
    Permissions
    Only the broker has rights in the cluster: the Role and RoleBinding colloq-runtime in this namespace. No ClusterRole, no access to nodes, Secrets, Deployments or other namespaces.
    ResourceVerbsWhy
    podsget, list, create, deleteRoom, personal-notebook and competition Pods. list is for the census: a restarted broker finds the running Pods by their labels and adopts them instead of creating them again.
    servicesget, list, create, deleteA ClusterIP Service for each room and its personal notebooks — the stable name the app reaches Jupyter by — and for the package-preparation proxy. A room deleted for good leaves a tombstone: a headless Service with no selector and no ClusterIP, so that its ID can't be opened again.
    pods/resizepatchChange a running room's memory and cores without a restart (Kubernetes 1.33+): the patch touches only memory and cpu of the kernel container. Optional: with runtime.inPlaceResize: false the rule is gone.
    persistentvolumeclaimsgetBefore a competition, check that the data volume is in place.
    networkpolicies (networking.k8s.io)getBefore a competition, check that the submissions' isolation policies exist. Without them submissions are refused, and the teacher sees why.
    There is no patch or update on Pods and Services: the broker doesn't edit live objects — it changes only a room's resources through pods/resize, and deletes and recreates everything else, with a UID precondition. RBAC can't constrain the fields of a Pod, so the Pod templates are fixed in the broker's code, and its HTTP API takes only a room ID, an environment from the catalog, an image revision, memory and cores: no arbitrary template, no hostPath, no image outside the catalog. Restricted Pod Security and your policies hold the rest.
    ServiceAccountAPI tokenWho
    colloq-appnoThe app. It never calls the Kubernetes API.
    colloq-runtimeyes: the broker's Pod mounts a projected token that the kubelet rotatesThe broker; bound to the Role above.
    colloq-kernelnoRoom and personal-notebook Pods: students' code holds no API credentials.
    the namespace defaultnoCompetition Pods: no token is mounted.
    All three ServiceAccounts set automountServiceAccountToken: false; only the broker's Pod turns the token on.
    
-
    Security
    Pod Security. Every Colloq Pod — the app, the broker, rooms, personal notebooks, competitions — passes the restricted profile: runAsNonRoot, uid and gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Only volumes are writable: emptyDir for /tmp, the home directory and /dev/shm (in memory: 64 MiB, 1 GiB for GPU rooms), and the PVCs. Your team puts the pod-security.kubernetes.io/enforce: restricted label on the namespace: the chart doesn't own it.
    Policy engines.
    • Requests and limits are set on every container, including the init container and the sidecar of competition Pods. A room kernel's requests equal its limits (QoS Guaranteed): memory and cores are reserved in full, plus 2 GiB of ephemeral storage.
    • Probes. The app: HTTP startupProbe and livenessProbe on /api/livez, readinessProbe on /api/readyz. The broker: TCP 8787. Room and personal-notebook kernels: TCP readiness and liveness probes on 8888; the liveness probe acts only after five minutes of refused connections, since a Pod with restartPolicy: Never ends with the class's variables when it does. Submission containers run to completion and have no probes; their exporter and the package proxy have readiness probes. If a probe rule doesn't fit run-to-completion Pods, exempt them by the label app.kubernetes.io/managed-by: colloq-runtime.
    • Labels. Chart objects carry app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm and helm.sh/chart. Pods the broker creates carry app.kubernetes.io/name, instance and part-of, app.kubernetes.io/managed-by: colloq-runtime and colloq.dev/role: kernel (the personal-notebook Pod also has colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. The app.kubernetes.io/* and colloq.* keys belong to the chart and the broker and can't be overridden.
    • Digests. The chart always renders the app and broker images as registry/path@sha256:… — never a tag, never latest. Kernel images in the catalog are accepted only in the same form: neither the chart nor the broker lets a tag-only reference through.
    • Registry. global.imageRegistry rewrites the registry of every image, including the catalog's kernels, the competition exporter and the package proxy.
    • Mutating webhooks. After creating a room's Pod the broker checks it against what it asked for. If a webhook changed its containers, volumes or security settings — injected a sidecar, rewrote imagePullPolicy — the broker refuses the Pod and the room doesn't start; the broker's log names the fields that changed. Exempt Pods labelled app.kubernetes.io/managed-by: colloq-runtime from such webhooks; for Istio, rooms.podLabels: {sidecar.istio.io/inject: "false"} is enough. Placement the cluster fills in by itself — PodNodeSelector, PodTolerationRestriction, priority, RuntimeClass, Kueue — doesn't disturb the check.
    What code runs as whom.
    PodImage and codeAPI tokenVolumesNetwork
    Appcolloq-app: the Colloq server on Node.jsnocolloq-data and colloq-workspacein from the ingress controller; out to the broker, rooms, DNS and, optionally, the AI model and a proxy
    Brokercolloq-runtime: the broker on Node.jsyesnonein from the app; out to the Kubernetes API, rooms, submissions and DNS
    Roomthe room environment's colloq-kernel: Jupyter and students' codenoits own folder of colloq-workspacein from the app and the broker on 8888; out: DNS only
    Personal notebookscolloq-kernel: students' code in personal notebooks, no GPUnoits room's folder, at the same pathas a room
    Submissioncolloq-kernel: the entrant's notebook or the metric; colloq-runtime: the result exporternofolders of colloq-data, inputs read-onlynone; in: only the broker to the exporter on 8765
    Package preparationcolloq-kernel: pip downloads entrants' own packagesnofolders of colloq-dataonly to its own proxy on 3128, and DNS
    Package proxycolloq-runtime: a CONNECT proxynononein from preparation; out: DNS and the package index — public PyPI on 443 or your mirror
    Isolation between rooms. Each room has a Pod of its own: its own processes, its own emptyDir, its own Jupyter token — the broker derives it from the room secret with an HMAC and gives it to the app only. The Pod mounts only its own folder of the colloq-workspace volume (subPath); other rooms' files aren't in it. The network policy lets only the app and the broker reach a room's port 8888 and lets nothing out of a room but DNS: neighbouring rooms, the app and the API are out of reach of students' code. Inside a room, participants share the kernel, the files and the Linux user — by design: it is a shared workspace. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node: test it from a room (see the checks after installing) and close it with your CNI's host policies if needed.
    Secrets.
    SecretWhat it holdsWho reads it
    config.existingSecret or colloq-app-configKeys are app settings: OPENAI_API_KEY, SESSION_SECRET, a proxy with a password in HTTPS_PROXY, or any other settingthe app
    colloq-runtime-auth, key runtime-tokenThe Bearer token the app uses with the brokerthe app and the broker
    colloq-room-secret, key room-secretThe key the broker derives rooms' Jupyter tokens and submission export tokens fromthe broker only
    colloq-metrics, key metrics-tokenThe Bearer token for /metrics, when metrics are onthe app and Prometheus
    global.imagePullSecretsThe mirror's .dockerconfigjson; broker Pods get the first one toothe kubelet
    Each can come from an existing Secret — for example one External Secrets creates from Vault. When the broker token, the room secret or the metrics token are not given, the chart generates them: under the helm CLI once, reading them back from the cluster with lookup afterwards. ArgoCD and other tools that render the chart with helm template can't read them back and would generate new values on every sync, restarting rooms — set existingSecret there. The broker token and the room secret are 43 to 128 characters of [A-Za-z0-9_-] from at least 32 random bytes: openssl rand -hex 32. Keep the secrets stable: a new SESSION_SECRET invalidates every link and sign-in handed out, and a new room secret replaces room Pods at their next start, losing Python variables. Without SESSION_SECRET, the app generates one once and keeps it in the colloq-data volume. When you move from a generated secret to existingSecret, copy its value into the new Secret first: the chart deletes its own.
    Secrets live in the colloq-data volume too: the setup token, the teachers' sign-in keys in the database, the model key if it was entered in the panel. Keep snapshots of that volume as private as Secrets.
    
+
    Security
    Pod Security. Every Colloq Pod — the app, the broker, rooms, personal notebooks, competitions — passes the restricted profile: runAsNonRoot, uid and gid 1000, seccomp RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true. Only volumes are writable: emptyDir for /tmp, the home directory and /dev/shm (in memory: 64 MiB, 1 GiB for GPU rooms, half the memory of a GPU submission and at most gpu.competitions.shmMb), and the PVCs. Your team puts the pod-security.kubernetes.io/enforce: restricted label on the namespace: the chart doesn't own it.
    Policy engines.
    • Requests and limits are set on every container, including the init container and the sidecar of competition Pods. A room kernel's requests equal its limits (QoS Guaranteed): memory and cores are reserved in full, plus 2 GiB of ephemeral storage.
    • Probes. The app: HTTP startupProbe and livenessProbe on /api/livez, readinessProbe on /api/readyz. The broker: TCP 8787. Room and personal-notebook kernels: TCP readiness and liveness probes on 8888; the liveness probe acts only after five minutes of refused connections, since a Pod with restartPolicy: Never ends with the class's variables when it does. Submission containers run to completion and have no probes; their exporter and the package proxy have readiness probes. If a probe rule doesn't fit run-to-completion Pods, exempt them by the label app.kubernetes.io/managed-by: colloq-runtime.
    • Labels. Chart objects carry app.kubernetes.io/name: colloq, instance, component, version, part-of, managed-by: Helm and helm.sh/chart. Pods the broker creates carry app.kubernetes.io/name, instance and part-of, app.kubernetes.io/managed-by: colloq-runtime and colloq.dev/role: kernel (the personal-notebook Pod also has colloq.dev/kernel: own), competition-job, competition-resolver, competition-proxy. The app.kubernetes.io/* and colloq.* keys belong to the chart and the broker and can't be overridden.
    • Digests. The chart always renders the app and broker images as registry/path@sha256:… — never a tag, never latest. Kernel images in the catalog are accepted only in the same form: neither the chart nor the broker lets a tag-only reference through.
    • Registry. global.imageRegistry rewrites the registry of every image, including the catalog's kernels, the competition exporter and the package proxy.
    • Mutating webhooks. After creating a room's Pod the broker checks it against what it asked for. If a webhook changed its containers, volumes or security settings — injected a sidecar, rewrote imagePullPolicy — the broker refuses the Pod and the room doesn't start; the broker's log names the fields that changed. Exempt Pods labelled app.kubernetes.io/managed-by: colloq-runtime from such webhooks; for Istio, rooms.podLabels: {sidecar.istio.io/inject: "false"} is enough. Placement the cluster fills in by itself — PodNodeSelector, PodTolerationRestriction, priority, RuntimeClass, Kueue — doesn't disturb the check.
    What code runs as whom.
    PodImage and codeAPI tokenVolumesNetwork
    Appcolloq-app: the Colloq server on Node.jsnocolloq-data and colloq-workspacein from the ingress controller; out to the broker, rooms, DNS and, optionally, the AI model and a proxy
    Brokercolloq-runtime: the broker on Node.jsyesnonein from the app; out to the Kubernetes API, rooms, submissions and DNS
    Roomthe room environment's colloq-kernel: Jupyter and students' codenoits own folder of colloq-workspacein from the app and the broker on 8888; out: DNS only
    Personal notebookscolloq-kernel: students' code in personal notebooks, no GPUnoits room's folder, at the same pathas a room
    Submissioncolloq-kernel: the entrant's notebook or the metric; colloq-runtime: the result exporternofolders of colloq-data, inputs read-onlynone; in: only the broker to the exporter on 8765
    Package preparationcolloq-kernel: pip downloads entrants' own packagesnofolders of colloq-dataonly to its own proxy on 3128, and DNS
    Package proxycolloq-runtime: a CONNECT proxynononein from preparation; out: DNS and the package index — public PyPI on 443 or your mirror
    Isolation between rooms. Each room has a Pod of its own: its own processes, its own emptyDir, its own Jupyter token — the broker derives it from the room secret with an HMAC and gives it to the app only. The Pod mounts only its own folder of the colloq-workspace volume (subPath); other rooms' files aren't in it. The network policy lets only the app and the broker reach a room's port 8888 and lets nothing out of a room but DNS: neighbouring rooms, the app and the API are out of reach of students' code. Inside a room, participants share the kernel, the files and the Linux user — by design: it is a shared workspace. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node: test it from a room (see the checks after installing) and close it with your CNI's host policies if needed.
    Secrets.
    SecretWhat it holdsWho reads it
    config.existingSecret or colloq-app-configKeys are app settings: OPENAI_API_KEY, SESSION_SECRET, a proxy with a password in HTTPS_PROXY, or any other settingthe app
    colloq-runtime-auth, key runtime-tokenThe Bearer token the app uses with the brokerthe app and the broker
    colloq-room-secret, key room-secretThe key the broker derives rooms' Jupyter tokens and submission export tokens fromthe broker only
    colloq-metrics, key metrics-tokenThe Bearer token for /metrics, when metrics are onthe app and Prometheus
    global.imagePullSecretsThe mirror's .dockerconfigjson; broker Pods get the first one toothe kubelet
    Each can come from an existing Secret — for example one External Secrets creates from Vault. When the broker token, the room secret or the metrics token are not given, the chart generates them: under the helm CLI once, reading them back from the cluster with lookup afterwards. ArgoCD and other tools that render the chart with helm template can't read them back and would generate new values on every sync, restarting rooms — set existingSecret there. The broker token and the room secret are 43 to 128 characters of [A-Za-z0-9_-] from at least 32 random bytes: openssl rand -hex 32. Keep the secrets stable: a new SESSION_SECRET invalidates every link and sign-in handed out, and a new room secret replaces room Pods at their next start, losing Python variables. Without SESSION_SECRET, the app generates one once and keeps it in the colloq-data volume. When you move from a generated secret to existingSecret, copy its value into the new Secret first: the chart deletes its own.
    Secrets live in the colloq-data volume too: the setup token, the teachers' sign-in keys in the database, the model key if it was entered in the panel. Keep snapshots of that volume as private as Secrets.
    
 
    Network
    Every flow Colloq needs. The chart declares them as network policies, ingress and egress, so the installation works in a default-deny namespace too.
    FromToPortWhy
    Ingress controllerAppTCP 3000All user traffic: pages, API, WebSocket, server-sent events. networkPolicy.ingressController.from says who is admitted (by default the ingress-nginx Pods in the ingress-nginx namespace). A controller on the host network arrives from node addresses: give an ipBlock of your nodes then.
    AppBrokerTCP 8787Start, resize and stop a room's Pod, start a submission. Requests carry a Bearer token.
    AppRoom and personal-notebook PodsTCP 8888Jupyter: running cells, the terminal. With the room's token.
    BrokerRoom and personal-notebook PodsTCP 8888Check that Jupyter is up and accepts the token.
    BrokerSubmission and package-preparation PodsTCP 8765Collect the result from the exporter.
    BrokerKubernetes API server443 and 6443Create and delete Pods and Services. The network policy sees the address after DNAT, that is the API servers' own addresses, so by default any address on these ports is allowed. Narrow networkPolicy.kubeApiServer.cidrs to the output of kubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes. On Cilium the API server has an identity of its own that an ipBlock doesn't match: turn on networkPolicy.kubeApiServer.ciliumEntity, and the chart adds a CiliumNetworkPolicy in this namespace.
    Every Colloq Pod except submissionsCluster DNSUDP and TCP 53Service names. networkPolicy.dns.to says where DNS is (by default k8s-app: kube-dns in kube-system). Rooms can lose DNS too: rooms.network: none.
    Package preparationIts own package proxyTCP 3128pip goes only through a proxy that admits one package index.
    Package proxyPyPI mirror or public PyPI443 or the mirror's portIndex and files of competition entrants' own packages. A policy knows addresses, not names: the mirror's addresses and ports go into dependencies.mirrorEgress. In a closed network turn off the general rule for public addresses on 443: dependencies.publicEgress: false.
    AppAI model, outbound proxy, GitHubthe model's or proxy's portOptional: the Oracle and notebook import from GitHub. By default the app may go nowhere outside: addresses and ports go into networkPolicy.app.egress.
    PrometheusAppTCP 3000, /metricsOptional, with metrics.enabled: networkPolicy.monitoring.from says who is admitted.
    Nodes (kubelet)Registry mirrorTCP 443Images. Not described by network policies.
    If the cluster runs NodeLocal DNSCache, DNS queries go to a node-local address rather than to the kube-dns Pods: add that address to networkPolicy.dns.to, for example ipBlock: {cidr: 169.254.20.10/32}.
    Students' code doesn't get out: room and personal-notebook Pods have no egress but DNS, and rooms.network: none takes that away too. So pip install in a cell doesn't work: libraries come in kernel images; see Images and the registry mirror. Competition submissions get no network at all, not even DNS, and that is checked before any code runs: an init container tries to get out and lets the submission run only after three refusals in a row; if the network answers, the submission fails.
    The client address: TRUSTED_PROXIES=private. Per-address caps — 60 newcomers to a room per 10 minutes, 20 Oracle questions a minute, competition sign-ins and joins — must see the student's address, not the ingress controller's. The app believes X-Forwarded-For only from connections whose address is in TRUSTED_PROXIES; the chart sets private (config.inbound.trustedProxies) — every private range, wherever the controller's Pod comes from. That is safe because only the ingress controller is admitted to the app's port 3000: the network policy lets no one else in, students' code included, so nobody inside the cluster can forge the header. The client is the rightmost address in the header that is not itself trusted; when every hop is trusted — and on a corporate network users' addresses are private too — the leftmost stands, which is the one the controller wrote. So the controller must replace the header rather than append its own address to one the client sent: ingress-nginx does that by default, while use-forwarded-headers and compute-full-forwarded-for are off. If a load balancer in front of the controller rewrites the source address, every student looks like one address: preserve the client address (externalTrafficPolicy: Local on the controller's Service, or PROXY protocol). If many people come from one address — through a NAT or a VPN concentrator — list it in config.inbound.sharedAddresses: per-address caps don't apply to it, while per-room and per-person caps still do.
    ingress-nginx. The chart puts these annotations on its Ingress (ingress.annotations; a key set to null drops an annotation):
    nginx.ingress.kubernetes.io/proxy-body-size: 256m       # competition data: up to 200 MB in one request
 nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"  # WebSocket and server-sent events live for hours
 nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
@@ -37,7 +37,7 @@ 
    Sign-in through your SSO
    If a proxy in front of Colloq check
       - cidrs: [10.20.0.15/32]              # the Teleport proxy, for the keys
         ports: [443]
    Instead of fetching the keys over the network, put the JWKS document in config.sso.jwks: the chart mounts it from the ConfigMap colloq-sso-jwks and no egress is needed, but a key rotation in Teleport needs a new value. The header is Teleport-Jwt-Assertion by default; config.sso.header sets another. Authorization won't do: the browser sends the room token in it, and a proxy writing its own there would break files, history and the Oracle. oauth2-proxy with --pass-access-token passes the token in X-Forwarded-Access-Token — fine when the IdP issues access tokens as JWTs (Keycloak does); the audience is then whatever the IdP writes into the access token.
    Token fields. By default: who it is — sub or username; the name — name, traits.name, traits.full_name, traits.display_name, username; the email — email, traits.email, traits.mail, username; roles — roles, groups, traits.groups. config.sso.claims.* sets other paths, and the first non-empty one wins. If the token carries only a login, config.sso.emailDomain turns it into the address teachers are recognised by. A line at start says where the keys come from and whom Colloq adds; a refused token is logged once a minute with the reason.
    Students without an account. Only someone with an account gets through Teleport. If students have none, keep the ordinary way in for them: an Ingress to the same Service at a public address and the class links, with config.publicUrl set to that public address, which links are built from. Teachers then open both the panel and the rooms at the Teleport address — the panel still builds the links for students from the public one: the token is checked the same way wherever the request came from, and a page at the Teleport address passes the origin check when the Teleport agent is in config.inbound.trustedProxies (the chart sets private).
    What stays with the proxy. A token is a pass until it expires: if Colloq can also be reached around the proxy, a stolen token works until its exp. Keep the token lifetime short, and keep the token header out of access logs.
    
 
    Storage
    PVCWhat it holdsWho mounts itDefault size
    colloq-dataThe SQLite database (colloq.db with its WAL journal), the link-signing key, the setup token, output images, class page files in page-files/, competition data, hidden answers and submissions, package sets, database snapshots in snapshots/the app; competition Pods — their own folders10 GiB; more for competitions with large data, and about 0.3–0.8 GiB for each 33-class course with class pages
    colloq-workspaceRoom files, a <room>/ folder eachthe app; a room's Pod and its personal-notebook Pod — only that room's folder100 GiB
    The storage class, size and access mode are set for both volumes at once (persistence.storageClass, persistence.accessMode) or for each (persistence.data.*, persistence.workspace.*); existingClaim attaches a PVC you already have. With persistence.retain: true, the default, the PVCs survive both helm uninstall and deleting the ArgoCD application.
    ReadWriteMany: preferred. With RWX on both volumes, room Pods spread over the cluster wherever the scheduler puts them, and the installation's capacity is bounded by the quota rather than by one node. CephFS or NFSv4, where the protocol itself handles file locking, will do; SQLite has a single writer, the app. SQLite's own documentation warns that locking is unreliable on many NFS implementations: if in doubt, keep the database volume on block storage (persistence.data.accessMode: ReadWriteOnce), which brings back running every Pod with a volume on one node.
    ReadWriteOnce: one node. This is the chart's default. An RWO volume attaches to one node, so everything that mounts Colloq's volumes must run there: if at least one of the volumes is RWO, the chart sets RUNTIME_COLOCATE_WITH_APP=1 for the broker, and every Pod that mounts a volume — rooms, personal notebooks, submissions, package preparation — gets a required podAffinity to the app's Pod (label colloq.dev/role: app, topology kubernetes.io/hostname). The app's own Pod prefers the node where rooms already run: after an upgrade the new Pod needs exactly the node the volume is attached to. The broker mounts no volume and can run anywhere. What follows: the whole installation's capacity is one node's free resources; a room beyond them doesn't start, and after 20 seconds of waiting the teacher sees what was short — memory, cores or a GPU; maintenance on that node is a break for all of Colloq. Pick a node with headroom; see Resources and quotas.
    File ownership. Every Pod writes as uid and gid 1000. The app's Pod sets fsGroup: 1000, and a fresh block volume becomes writable for group 1000 by itself. For RWX volumes the CSI driver doesn't apply fsGroup by default (fsGroupPolicy: ReadWriteOnceWithFSType covers only RWO volumes with a filesystem type): the root of such a volume must be writable by uid 1000 on the storage side.
    No per-room disk quota. The colloq-workspace volume is shared: a student who fills it stops file saving for everyone. Watch PVC usage — the kubelet metrics kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes, or colloq_disk_free_bytes from /metrics — with an alert at 80 %. If the StorageClass allows volume expansion, the PVCs can grow.
    
-
    Images and the registry mirror
    ImageWhat it is
    ghcr.io/colloq-edu/colloq-app:vX.Y.ZThe app.
    ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZThe broker; also the submissions' result exporter and the package proxy.
    ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseRoom and submission kernels: the base and kaggle-base environments, CPU only.
    Every image is built for amd64 only. A release and its chart share one version number. In the published chart the defaults carry the digest of every image of the release: the app's and the broker's in image.app.digest and image.runtime.digest, the kernels' in the environment catalog catalog.environments. The same values come as a file of their own, values-X.Y.Z.yaml, attached to the release on GitHub: handy for copying the images into a mirror and for diffing releases in a GitOps repository. The chart in the source tree carries no digests and refuses to render without them.
    The environment chosen for a class is a catalog entry: a name, an image by digest, a GPU flag. An environment of your own — other libraries, or a GPU — is built outside the cluster from the repository's kernel/ directory with KERNEL_ENV=<name>, pushed to your registry and added to catalog.environments: on Kubernetes the panel doesn't build environments. The base-gpu image (about 9 GB of CUDA wheels) is not published; build a GPU environment the same way.
    The registry mirror. global.imageRegistry replaces the registry in every image reference — the app's, the broker's, the catalog's kernels, the competition exporter and package proxy — and keeps the path and the digest. The value may carry a path too: a Harbor proxy cache of ghcr.io in a project called ghcr works as is, and harbor.corp.example/ghcr gives harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. So the mirror must hold the images under the same paths. Pull secrets go into global.imagePullSecrets: the app's and the broker's Pods get them, and every Pod the broker creates gets the first one.
    Without internet access. On a machine that can reach ghcr.io:
    # the release's chart
+
    Images and the registry mirror
    ImageWhat it is
    ghcr.io/colloq-edu/colloq-app:vX.Y.ZThe app.
    ghcr.io/colloq-edu/colloq-runtime:vX.Y.ZThe broker; also the submissions' result exporter and the package proxy.
    ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base, …-kaggle-baseRoom and submission kernels: the base and kaggle-base environments, CPU only.
    Every image is built for amd64 only. A release and its chart share one version number. In the published chart the defaults carry the digest of every image of the release: the app's and the broker's in image.app.digest and image.runtime.digest, the kernels' in the environment catalog catalog.environments. The same values come as a file of their own, values-X.Y.Z.yaml, attached to the release on GitHub: handy for copying the images into a mirror and for diffing releases in a GitOps repository. The chart in the source tree carries no digests and refuses to render without them.
    The environment chosen for a class is a catalog entry: a name, an image by digest, a GPU flag. An environment of your own — other libraries, or a GPU — is built outside the cluster from the repository's kernel/ directory with KERNEL_ENV=<name>, pushed to your registry and added to catalog.environments: on Kubernetes the panel doesn't build environments. The GPU images, base-gpu (torch 2.14.1 and torchvision as CUDA 13.0 wheels, an image of about 9.9 GB) and kaggle-gpu on top of it (the kaggle-base set plus OpenCV and timm, about 2.8 GB more, 12.7 GB in all), are not published; build them the same way, the parent first. The nodes that run them on a card need an NVIDIA driver of 580 or newer. Every Pod of a GPU image, the metric's and package preparation's on the students' nodes included, has gpu.competitions.pullSeconds (900 seconds) to pull it onto a node that has never run it, before the job's own time starts; to spare the first class that wait, pull the GPU images onto those nodes in advance, for instance with a DaemonSet whose init containers are those images, each running true.
    The registry mirror. global.imageRegistry replaces the registry in every image reference — the app's, the broker's, the catalog's kernels, the competition exporter and package proxy — and keeps the path and the digest. The value may carry a path too: a Harbor proxy cache of ghcr.io in a project called ghcr works as is, and harbor.corp.example/ghcr gives harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:…. So the mirror must hold the images under the same paths. Pull secrets go into global.imagePullSecrets: the app's and the broker's Pods get them, and every Pod the broker creates gets the first one.
    Without internet access. On a machine that can reach ghcr.io:
    # the release's chart
 helm pull oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z
 
 # every image it refers to, the catalog's kernels included
@@ -161,7 +161,7 @@ 
    Installation
    Your team creates the namespace with its Po
         print(host, port, "OPEN: check the network policies")
     except OSError as error:
         print(host, port, "closed:", error)
  • If you use competitions: a trial submission through to the end; leaderboard rows should appear by themselves, with no page reload.
  • Before the first group: a restore from a backup into a separate namespace.
    • 
-
    Chart values
    The main values and the variables they set. The full list, each value with an explanation, is the chart's values.yaml: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Other app settings from the Settings table of the VM guide go into config.extraEnv, or as keys of the Secret named in config.existingSecret.
    ValueVariable and what it setsDefault
    Address and ingress
    ingress.hostThe hostname; required while the Ingress is on.—
    config.publicUrlPUBLIC_URL — the external address links are built from.https://<ingress.host>
    ingress.className, ingress.annotationsThe controller class and annotations.nginx, the annotations under Network
    ingress.tls.secretName, ingress.tls.certManager.*The certificate's Secret, or a cert-manager issuer.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — whose X-Forwarded-For to believe.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — NAT and VPN addresses that per-address caps don't apply to.empty
    config.inbound.hstsHSTS — the app's Strict-Transport-Security header.true
    Images and the catalog
    global.imageRegistryA registry, optionally with a path, instead of ghcr.io for every image, the kernel catalog included.empty
    global.imagePullSecretsSecrets for pulling images; broker Pods get the first one (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestDigests of the app and broker images.the release's digests
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — the environment catalog: a name, an image by digest, GPU; several revisions of one name with exactly one current: true.the release's base and kaggle-base
    commonLabelsLabels on every chart object and every broker Pod (part of RUNTIME_POD_LABELS).—
    Secrets
    config.existingSecretAn existing Secret whose keys become the app's variables.—
    secrets.runtimeToken.*, secrets.roomSecret.*The broker token and the room secret: existingSecret and existingSecretKey, or value.generated once by the chart (helm CLI only)
    Storage
    persistence.accessModeThe volumes' access mode. If either volume is ReadWriteOnce, the broker gets RUNTIME_COLOCATE_WITH_APP=1: every Pod with a volume runs on the app's node.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeThe storage class and volume sizes; existingClaim attaches a PVC you have.the cluster default; 10Gi and 100Gi
    persistence.retainKeep the PVCs when the release or the ArgoCD application is deleted.true
    Rooms
    rooms.memoryRUNTIME_KERNEL_MEMORY — a room's memory unless the class sets its own.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — a room's memory ceiling. Set it explicitly: without it the broker takes its own node's memory minus 1 GiB.the broker node's memory − 1 GiB
    rooms.cpuRUNTIME_KERNEL_CPU — a room's cores unless the class sets its own.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — space for a room's /tmp and home directory.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — with none, rooms get no DNS either.DNS only
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — where room and personal-notebook Pods run; competition Pods don't get them.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — the priority class of every Pod the broker creates.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — labels and annotations on every broker Pod; the chart adds app.kubernetes.io/name, instance and part-of to the labels itself.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — with false, live rooms aren't resized and the Role has no pods/resize rule.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — the RuntimeClass of GPU rooms; empty means none.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — added for GPU rooms.—
    Network and outbound
    networkPolicy.ingressController.fromWho is admitted to the app's port 3000.ingress-nginx
    networkPolicy.dns.toWhere cluster DNS is.kube-dns in kube-system
    networkPolicy.kubeApiServer.*The API servers' addresses and ports for the broker; ciliumEntity for Cilium.any address on 443 and 6443
    networkPolicy.app.egressWhere the app may go outside: the model, a proxy, GitHub.nowhere
    networkPolicy.monitoring.fromWho scrapes /metrics.Prometheus in monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — the app's outbound proxy. Service names (.svc, single-label names) and private addresses bypass it by themselves.—
    outbound.extraCa.pem or .existingConfigMapNODE_EXTRA_CA_CERTS — the internal CA's certificates as PEM, for the app and package preparation.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — the PyPI mirror for competition entrants' own packages: https, no credentials.public PyPI
    dependencies.mirrorEgress, dependencies.publicEgressThe mirror's addresses and ports for the package proxy; the general rule for public addresses on 443.—; true
    Classes and the Oracle
    config.timezoneTZ — the time zone: day boundaries for daily quotas, dates.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — the interface language until the owner chooses one in the panel.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — the limit for one file and for all uploads to a room.50 and 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — the Oracle's model; for a model on the internal network, vllm, ollama or custom. The key is better kept in config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvOther app settings, as NAME: value.—
    Monitoring, backups, resources
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — /metrics behind a Bearer token, and a ServiceMonitor that uses the same token.off
    config.logFormatLOG_FORMAT — with json, one JSON object per line.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — how often to write a consistent database snapshot into /data/snapshots, and how many to keep.24 h and 7
    app.resources, runtime.resourcesRequests and limits of the app and the broker.see Resources and quotas
    
+
    Chart values
    The main values and the variables they set. The full list, each value with an explanation, is the chart's values.yaml: helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z. Other app settings from the Settings table of the VM guide go into config.extraEnv, or as keys of the Secret named in config.existingSecret.
    ValueVariable and what it setsDefault
    Address and ingress
    ingress.hostThe hostname; required while the Ingress is on.—
    config.publicUrlPUBLIC_URL — the external address links are built from.https://<ingress.host>
    ingress.className, ingress.annotationsThe controller class and annotations.nginx, the annotations under Network
    ingress.tls.secretName, ingress.tls.certManager.*The certificate's Secret, or a cert-manager issuer.colloq-tls
    config.inbound.trustedProxiesTRUSTED_PROXIES — whose X-Forwarded-For to believe.private
    config.inbound.sharedAddressesSHARED_ADDRESSES — NAT and VPN addresses that per-address caps don't apply to.empty
    config.inbound.hstsHSTS — the app's Strict-Transport-Security header.true
    Images and the catalog
    global.imageRegistryA registry, optionally with a path, instead of ghcr.io for every image, the kernel catalog included.empty
    global.imagePullSecretsSecrets for pulling images; broker Pods get the first one (RUNTIME_IMAGE_PULL_SECRET).—
    image.app.digest, image.runtime.digestDigests of the app and broker images.the release's digests
    catalog.environments, catalog.defaultEnvironmentKERNEL_CATALOG_FILE, RUNTIME_CATALOG_FILE — the environment catalog: a name, an image by digest, GPU; several revisions of one name with exactly one current: true.the release's base and kaggle-base
    commonLabelsLabels on every chart object and every broker Pod (part of RUNTIME_POD_LABELS).—
    Secrets
    config.existingSecretAn existing Secret whose keys become the app's variables.—
    secrets.runtimeToken.*, secrets.roomSecret.*The broker token and the room secret: existingSecret and existingSecretKey, or value.generated once by the chart (helm CLI only)
    Storage
    persistence.accessModeThe volumes' access mode. If either volume is ReadWriteOnce, the broker gets RUNTIME_COLOCATE_WITH_APP=1: every Pod with a volume runs on the app's node.ReadWriteOnce
    persistence.storageClass, persistence.data.size, persistence.workspace.sizeThe storage class and volume sizes; existingClaim attaches a PVC you have.the cluster default; 10Gi and 100Gi
    persistence.retainKeep the PVCs when the release or the ArgoCD application is deleted.true
    Rooms
    rooms.memoryRUNTIME_KERNEL_MEMORY — a room's memory unless the class sets its own.4Gi
    rooms.maxMemoryRUNTIME_KERNEL_MEMORY_MAX — a room's memory ceiling. Set it explicitly: without it the broker takes its own node's memory minus 1 GiB.the broker node's memory − 1 GiB
    rooms.cpuRUNTIME_KERNEL_CPU — a room's cores unless the class sets its own.2
    rooms.ephemeralStorageRUNTIME_KERNEL_EPHEMERAL — space for a room's /tmp and home directory.2Gi
    rooms.networkCOLLOQ_ROOM_NETWORK — with none, rooms get no DNS either.DNS only
    rooms.nodeSelector, rooms.tolerationsRUNTIME_ROOM_NODE_SELECTOR, RUNTIME_ROOM_TOLERATIONS — where room, personal-notebook and competition Pods run: submissions, the metric and package preparation (not its package proxy). A GPU submission's Pod gets gpu.nodeSelector and gpu.tolerations on top, so its node must match both.—
    rooms.priorityClassNameRUNTIME_PRIORITY_CLASS — the priority class of every Pod the broker creates.—
    rooms.podLabels, rooms.podAnnotationsRUNTIME_POD_LABELS, RUNTIME_POD_ANNOTATIONS — labels and annotations on every broker Pod; the chart adds app.kubernetes.io/name, instance and part-of to the labels itself.—
    runtime.inPlaceResizeRUNTIME_IN_PLACE_RESIZE — with false, live rooms aren't resized and the Role has no pods/resize rule.true
    gpu.runtimeClassNameRUNTIME_GPU_RUNTIME_CLASS — the RuntimeClass of GPU rooms and GPU submissions; empty means none.nvidia
    gpu.nodeSelector, gpu.tolerationsRUNTIME_GPU_NODE_SELECTOR, RUNTIME_GPU_TOLERATIONS — added for GPU rooms and GPU submissions.—
    gpu.competitions.resource, gpu.competitions.countRUNTIME_COMPETITION_GPU_RESOURCE, RUNTIME_COMPETITION_GPU_COUNT — what the notebook Pod of a competition with the GPU switch on requests: the device plugin's resource name, such as nvidia.com/gpu or nvidia.com/mig-1g.10gb, and how many units of it, 1 to 8. nvidia.com/gpu is what GPU rooms request too: with it, submissions take the rooms' cards; see GPU nodes. Empty: submissions get no GPU, and such a competition refuses uploads.empty, 1
    gpu.competitions.slotsRUNTIME_COMPETITION_GPU_SLOTS — how many GPU submissions the cluster runs at once for this installation, 1 to 256: the units set aside for submissions over count. The app's queue keeps that many at the broker and lets CPU work go past the rest; see GPU nodes.empty: one at a time
    gpu.competitions.shmMbRUNTIME_COMPETITION_GPU_SHM_MB — the ceiling of a GPU submission's in-memory /dev/shm, MiB, 0 to 65536; it is never more than half the submission's memory. 0: none, which MPS sharing needs.1024
    gpu.competitions.pullSecondsRUNTIME_COMPETITION_GPU_PULL_SECONDS — how long any Pod of a GPU catalog image may take to pull it onto its node before the job's own time starts, 0 to 3600 seconds: a submission, the metric and package preparation alike, with a GPU or without one.900
    Network and outbound
    networkPolicy.ingressController.fromWho is admitted to the app's port 3000.ingress-nginx
    networkPolicy.dns.toWhere cluster DNS is.kube-dns in kube-system
    networkPolicy.kubeApiServer.*The API servers' addresses and ports for the broker; ciliumEntity for Cilium.any address on 443 and 6443
    networkPolicy.app.egressWhere the app may go outside: the model, a proxy, GitHub.nowhere
    networkPolicy.monitoring.fromWho scrapes /metrics.Prometheus in monitoring
    outbound.httpsProxy, outbound.httpProxy, outbound.noProxyHTTPS_PROXY, HTTP_PROXY, NO_PROXY — the app's outbound proxy. Service names (.svc, single-label names) and private addresses bypass it by themselves.—
    outbound.extraCa.pem or .existingConfigMapNODE_EXTRA_CA_CERTS — the internal CA's certificates as PEM, for the app and package preparation.—
    dependencies.indexUrl, dependencies.filesHostsDEPENDENCY_INDEX_URL, DEPENDENCY_FILES_HOSTS — the PyPI mirror for competition entrants' own packages: https, no credentials.public PyPI
    dependencies.mirrorEgress, dependencies.publicEgressThe mirror's addresses and ports for the package proxy; the general rule for public addresses on 443.—; true
    Classes and the Oracle
    config.timezoneTZ — the time zone: day boundaries for daily quotas, dates.Europe/Moscow
    config.uiLanguageUI_LANGUAGE — the interface language until the owner chooses one in the panel.ru
    config.maxUploadMb, config.maxSessionMbMAX_UPLOAD_MB, MAX_SESSION_MB — the limit for one file and for all uploads to a room.50 and 1024
    config.ai.provider, .baseUrl, .model, .apiKeyAI_PROVIDER, OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY — the Oracle's model; for a model on the internal network, vllm, ollama or custom. The key is better kept in config.existingSecret.openai, https://api.openai.com/v1, gpt-4o-mini
    config.extraEnvOther app settings, as NAME: value.—
    Monitoring, backups, resources
    metrics.enabled, metrics.serviceMonitor.enabledMETRICS_TOKEN — /metrics behind a Bearer token, and a ServiceMonitor that uses the same token.off
    config.logFormatLOG_FORMAT — with json, one JSON object per line.text
    config.dbSnapshotHours, config.dbSnapshotKeepDB_SNAPSHOT_HOURS, DB_SNAPSHOT_KEEP — how often to write a consistent database snapshot into /data/snapshots, and how many to keep.24 h and 7
    app.resources, runtime.resourcesRequests and limits of the app and the broker.see Resources and quotas
    
 
    Resources and quotas
    The numbers come from the limits in the code and from the same reasoning as for one VM, turned here into Pod requests and limits. The main difference from a VM: a room's Pod reserves its memory and cores in full rather than sharing them with its neighbours.
    PodHow manyCPU: request / limitMemory: request / limitEphemeral storage
    App1250m / 2512Mi / 2Gi—
    Broker1100m / 1128Mi / 512Mi—
    Roomone per running room2 / 24Gi / 4Gi2Gi
    Personal notebooksone per room that uses themthe room's, or the class's own number, request equal to limit; about 250m per student is neededthe room's, or Memory per class; 1–1.5 GiB per student is needed2Gi
    Submissionone per slotthe submission's cores + 100m / + 500mthe submission's memory + 64Mi / + 256Mi128Mi / 320Mi
    Package preparationwhile it runs1100m / 1500m, proxy 100m / 500m2112Mi / 2304Mi, proxy 64Mi / 128Miup to 320Mi
    The minimum quota for one class at a time (a room of 2 cores and 4 GiB), in the ResourceQuota keys requests.cpu, requests.memory, limits.cpu, limits.memory and pods:
    ScenarioCPU requestsMemory requestsCPU limitsMemory limitsPods
    (a) 30 students, one shared notebook2.354.6 GiB56.5 GiB3
    (b) 30 students, each with a personal notebook (1.5 GiB and 0.25 core per student)9.8549.6 GiB12.551.5 GiB4
    (c) 30 students and a competition, 7 slots of 2 cores and 2 GiB17.0519.1 GiB22.522.25 GiB10
    (e) 100 people in a lecture room2.354.6 GiB56.5 GiB3
    A room keeps its Pod while anyone is in it and for 2 hours after everyone has left, so back-to-back classes overlap: for each further class within two hours, add another room. Competition slots are one queue for the whole installation. A quota for scenario (c) with headroom for the next class's room and for package preparation:
    apiVersion: v1
 kind: ResourceQuota
 metadata:
@@ -177,7 +177,7 @@ 
    Resources and quotas
    The numbers come from the limits in
     pods: "25"
     services: "100"
     persistentvolumeclaims: "2"
-    requests.storage: 150Gi
    • The API won't create a Pod over the quota, and the room or the submission gets a refusal. Keep the quota above the sum and the slot count within it: Executors at once in the Resources tab, or COMPETITION_SLOTS in config.extraEnv. The automatic count works from the cores and memory of the node the app runs on and knows nothing about your quota.
    • Services pile up: every room deleted for good leaves a headless Service without a ClusterIP, so give the Service count limit headroom.
    • A LimitRange with a per-container maximum must admit a room (2 cores and 4 GiB; a GPU room with its memory) and the largest submission you allow in competitions: 2 cores and 4 GiB by default.
    • With ReadWriteOnce the whole quota must fit on one node: for scenario (c), a node with at least 24 vCPU and 32 GiB free; for (b), at least 16 vCPU and 64 GiB.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core. This was measured on a VM, not in a cluster, but the process is the same: the chart's request of 512 MiB is twice the measured figure, and the limits of 2 cores and 2 GiB leave plenty of headroom.
    
+    requests.storage: 150Gi
    • The API won't create a Pod over the quota, and the room or the submission gets a refusal. Keep the quota above the sum and the slot count within it: Executors at once in the Resources tab, or COMPETITION_SLOTS in config.extraEnv. The automatic count works from the cores and memory of the node the app runs on and knows nothing about your quota. A GPU submission refused that way goes back to the queue and is retried instead. A quota that names the GPU resource, requests.nvidia.com/gpu, bounds GPU rooms and GPU submissions together: keep gpu.competitions.slots × gpu.competitions.count within it, less what GPU rooms hold.
    • Services pile up: every room deleted for good leaves a headless Service without a ClusterIP, so give the Service count limit headroom.
    • A LimitRange with a per-container maximum must admit a room (2 cores and 4 GiB; a GPU room with its memory) and the largest submission you allow in competitions: 2 cores and 4 GiB by default.
    • With ReadWriteOnce the whole quota must fit on one node: for scenario (c), a node with at least 24 vCPU and 32 GiB free; for (b), at least 16 vCPU and 64 GiB.
    The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core. This was measured on a VM, not in a cluster, but the process is the same: the chart's request of 512 MiB is twice the measured figure, and the limits of 2 cores and 2 GiB leave plenty of headroom.
    
 
    Operations
    Upgrades. helm upgrade colloq oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z -n colloq -f colloq-values.yaml, or a new targetRevision in ArgoCD. The app and the broker update with the Recreate strategy: the old Pod stops before the new one starts, because SQLite has one writer. The break lasts seconds to a minute; a start with a database migration is given up to 10 minutes. Open rooms reconnect by themselves. Room Pods survive the upgrade: the broker leaves them alone when it stops, the new one adopts them, and Python variables stay. Placement settings — rooms.nodeSelector, rooms.tolerations, the priority class, the storage mode — go to a room's next Pod, while a running one stays where it is. Still, upgrade between classes.
    Kernel revisions. A room stays on the kernel image revision it started with and runs only that one. On helm upgrade the chart keeps the previous release's revisions in the catalog with current: false by itself (at most catalog.maxRetainedPerEnvironment, three by default, per environment): it reads them from the live colloq-catalog ConfigMap, and rooms started earlier go on with their Python. ArgoCD renders without cluster access and cannot see the previous catalog: there a room whose revision is gone moves to its environment's current revision on its next start, and the app logs a line about it. To keep previous revisions under ArgoCD too, list them in catalog.environments with current: false. Keep the older revisions' images in the mirror while rooms use them.
    The data version. If a new release raises the database schema version, the app first writes a snapshot, /data/snapshots/pre-schema-<from>-to-<to>-<time>.db; such snapshots are never removed automatically. An older release won't start on a database with a newer schema: the log says "colloq.db … was written by a newer Colloq", and the Pod goes into CrashLoopBackOff. So a rollback in that case is the older chart together with restoring the database: stop the app, put the snapshot in place of colloq.db from a maintenance Pod and delete colloq.db-wal and colloq.db-shm, then install the older chart version. Anything done after the upgrade is lost. COLLOQ_ALLOW_SCHEMA_DOWNGRADE=1 in config.extraEnv starts the older release on the newer database without a restore — only if you know the migration is harmless for it.
    kubectl -n colloq scale deploy/colloq-app --replicas=0
 kubectl -n colloq apply -f - <<'EOF'
 apiVersion: v1
@@ -210,7 +210,7 @@ 
    Operations
    Upgrades. helm upgr
 kubectl -n colloq exec colloq-maintenance -- sh -c \
   'cp /data/snapshots/pre-schema-….db /data/colloq.db && rm -f /data/colloq.db-wal /data/colloq.db-shm'
 kubectl -n colloq delete pod colloq-maintenance
-helm rollback colloq <revision before the upgrade> -n colloq
    Backups. Snapshot both PVCs with your own tools — Velero with CSI snapshots, or your storage's snapshots — and the namespace's Secrets unless they come from Vault, in which case Vault is their backup. A snapshot of a live SQLite volume is a crash image: the database usually recovers from it, but it is not something to rely on. So every config.dbSnapshotHours hours (24 in the chart) the app itself writes a consistent snapshot, /data/snapshots/colloq-<time>.db, and keeps the last config.dbSnapshotKeep (7): every volume snapshot holds a whole copy of the database no older than a day. The volumes are snapshotted separately, so room files and the database in a backup can be minutes apart. Copies of the colloq-data volume contain the teachers' sign-in keys, the setup token and the model key if it was entered in the panel: keep them as secrets. Test a restore into a separate namespace at least once a term. The colloq-host backup and restore commands are for the VM and are not used in a cluster.
    Monitoring. The liveness probe is /api/livez, readiness is /api/readyz: it checks only the database and the working files, so a broker or API hiccup doesn't take the site out of rotation, and kernel trouble shows where code is run and in /api/health. With metrics.enabled, /metrics serves Prometheus text to requests carrying Authorization: Bearer <token>; without the setting it answers 404. Useful for alerts: free space on the volumes (colloq_disk_free_bytes), the time of the last database snapshot (colloq_db_snapshot_last_timestamp_seconds), notebook save and kernel start failures (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Logs go to stdout and stderr; with config.logFormat: json each line is a JSON object with time, level, msg and context fields. The log contains room addresses, and a room address is a sign-in link, so access to the log equals access to the rooms. Staff can use GET /api/instance/operations: queue age, retries, notebook save failures, memory reservations, free disk.
    GPU nodes. A GPU room requests nvidia.com/gpu: 1 and the RuntimeClass from gpu.runtimeClassName (nvidia by default; empty means none); gpu.nodeSelector and gpu.tolerations pick the nodes. The driver, the device plugin or GPU Operator and the RuntimeClass itself are cluster-scoped objects your team provides. One card per room, not shared, unless your device plugin splits cards itself. Personal notebooks and competition submissions never get a GPU. Build the GPU environment's image yourself; see Images and the registry mirror.
    Node maintenance. Room Pods are bare Pods without a controller and with emptyDir: kubectl drain removes them only with --force --delete-emptydir-data, and the cluster autoscaler won't free a node that runs them until you add the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" through rooms.podAnnotations. A room brings a deleted Pod back at its next run: notebooks and files are there, Python variables are not. With ReadWriteOnce, maintenance on the app's node is a break for the whole installation.
    Uninstalling. helm uninstall removes the chart's objects except the PVCs — with persistence.retain: true they stay — and leaves the Pods and Services the broker created: they don't belong to Helm. Delete them by label: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
-
    Honest limits
    • One app replica. The database is SQLite with one writer: there is no high availability, and an upgrade or a node failure interrupts work while the Pod restarts. To grow, run separate installations, one per namespace — for example one per department.
    • SSO only through a proxy. Colloq has no SSO, LDAP or second-factor sign-in of its own: a proxy that signs a JWT provides it — see Sign-in through your SSO. Without such a proxy the owner and teachers sign in with personal links, and the panel can be closed off at the ingress controller — for example oauth2-proxy with the corporate IdP through ingress-nginx's external authentication annotations, on a separate Ingress for the paths /admin and /api/admin. Competition entrants sign in with their own key even behind the proxy.
    • A room link is a shared link. Whoever has it can join and, by default, run code; it can't be revoked. Rooms don't suit individually graded or confidential work: participants share the kernel, the files and the Linux user.
    • ReadWriteOnce means one node. The whole installation fits on one node, and its maintenance is a break for everyone.
    • Students' code has no internet. pip install in a cell doesn't work; libraries come only in the catalog's kernel images, built outside the cluster.
    • No per-room disk quota: the colloq-workspace volume is shared.
    • Network policies are not all of the isolation. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node, and Pods share the node's Linux kernel.
    • amd64 only.
    • Kernels on Debian 12. A scanner finds critical findings in them that Debian has not fixed; see “Images and the registry mirror”.
    • A new path. The chart is new in this release: test the installation in a non-production namespace before running classes in it.
    
+helm rollback colloq <revision before the upgrade> -n colloq
    Backups. Snapshot both PVCs with your own tools — Velero with CSI snapshots, or your storage's snapshots — and the namespace's Secrets unless they come from Vault, in which case Vault is their backup. A snapshot of a live SQLite volume is a crash image: the database usually recovers from it, but it is not something to rely on. So every config.dbSnapshotHours hours (24 in the chart) the app itself writes a consistent snapshot, /data/snapshots/colloq-<time>.db, and keeps the last config.dbSnapshotKeep (7): every volume snapshot holds a whole copy of the database no older than a day. The volumes are snapshotted separately, so room files and the database in a backup can be minutes apart. Copies of the colloq-data volume contain the teachers' sign-in keys, the setup token and the model key if it was entered in the panel: keep them as secrets. Test a restore into a separate namespace at least once a term. The colloq-host backup and restore commands are for the VM and are not used in a cluster.
    Monitoring. The liveness probe is /api/livez, readiness is /api/readyz: it checks only the database and the working files, so a broker or API hiccup doesn't take the site out of rotation, and kernel trouble shows where code is run and in /api/health. With metrics.enabled, /metrics serves Prometheus text to requests carrying Authorization: Bearer <token>; without the setting it answers 404. Useful for alerts: free space on the volumes (colloq_disk_free_bytes), the time of the last database snapshot (colloq_db_snapshot_last_timestamp_seconds), notebook save and kernel start failures (colloq_notebook_save_failures_total, colloq_kernel_start_failures_total). Logs go to stdout and stderr; with config.logFormat: json each line is a JSON object with time, level, msg and context fields. The log contains room addresses, and a room address is a sign-in link, so access to the log equals access to the rooms. Staff can use GET /api/instance/operations: queue age, retries, notebook save failures, memory reservations, free disk.
    GPU nodes. A GPU room requests nvidia.com/gpu: 1 and the RuntimeClass from gpu.runtimeClassName (nvidia by default; empty means none); gpu.nodeSelector and gpu.tolerations pick the nodes. The driver, the device plugin or GPU Operator and the RuntimeClass itself are cluster-scoped objects your team provides. The shipped GPU environments need a driver of 580 or newer: base-gpu and kaggle-gpu carry torch 2.14.1 as CUDA 13.0 wheels, which import on an older driver but never see the card. One card per room, not shared, unless your device plugin splits cards itself. Personal notebooks never get a GPU. A competition submission gets one only when its competition's GPU switch is on and gpu.competitions.resource is set: its notebook Pod, never the metric's, requests gpu.competitions.count units of that resource, with the rooms' nodeSelector and tolerations and, on top of them, the GPU rooms' RuntimeClass, nodeSelector and tolerations, and an in-memory /dev/shm of half its memory, at most gpu.competitions.shmMb (1024 MiB). What one unit is — a whole card, a time-sliced or MPS replica, a MIG slice — is your device plugin's configuration, and Colloq caps no video memory here itself. With MPS sharing, set gpu.competitions.shmMb: 0, because the device plugin mounts its MPS daemon's shared memory at /dev/shm, a volume of ours there would hide it, and CUDA would not start in the submission; and keep GPU rooms off MPS nodes, which renamed replicas (below) do, since a room keeps its own 1 GiB /dev/shm. Without the value, a competition with the switch on refuses uploads and says why. How many GPU submissions go to the cluster at once is gpu.competitions.slots, one while it is unset: the app's queue keeps that many at the broker, and CPU submissions go past the rest; once the cluster has sent one back for want of a unit, the queue takes no GPU submission until that one's retry is due or one of the running GPU submissions ends. A GPU submission the scheduler cannot place, or whose Pod the API server refuses — a ResourceQuota used up, a RuntimeClass the cluster lacks, an admission policy — goes back to the queue and is retried a few seconds to a minute later, and its teacher sees the cause. With nvidia.com/gpu, the resource GPU rooms request, submissions take units from the rooms' cards: a submission holds its unit for up to the competition's time limit, and a GPU room that finds no card free is refused after 20 seconds. Where classes use GPU rooms, give submissions a resource of their own, so the two never take the same cards: time-sliced replicas renamed to nvidia.com/gpu.shared (renameByDefault: true) on nodes labelled for that device-plugin configuration, or a MIG profile such as nvidia.com/mig-1g.10gb. Build the GPU environments' images yourself; see Images and the registry mirror.
    Node maintenance. Room Pods are bare Pods without a controller and with emptyDir: kubectl drain removes them only with --force --delete-emptydir-data, and the cluster autoscaler won't free a node that runs them until you add the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" through rooms.podAnnotations. A room brings a deleted Pod back at its next run: notebooks and files are there, Python variables are not. With ReadWriteOnce, maintenance on the app's node is a break for the whole installation.
    Uninstalling. helm uninstall removes the chart's objects except the PVCs — with persistence.retain: true they stay — and leaves the Pods and Services the broker created: they don't belong to Helm. Delete them by label: kubectl -n colloq delete pod,svc -l app.kubernetes.io/managed-by=colloq-runtime.
    
+
    Honest limits
    • One app replica. The database is SQLite with one writer: there is no high availability, and an upgrade or a node failure interrupts work while the Pod restarts. To grow, run separate installations, one per namespace — for example one per department.
    • SSO only through a proxy. Colloq has no SSO, LDAP or second-factor sign-in of its own: a proxy that signs a JWT provides it — see Sign-in through your SSO. Without such a proxy the owner and teachers sign in with personal links, and the panel can be closed off at the ingress controller — for example oauth2-proxy with the corporate IdP through ingress-nginx's external authentication annotations, on a separate Ingress for the paths /admin and /api/admin. Competition entrants sign in with their own key even behind the proxy.
    • A room link is a shared link. Whoever has it can join and, by default, run code; it can't be revoked. Rooms don't suit individually graded or confidential work: participants share the kernel, the files and the Linux user.
    • ReadWriteOnce means one node. The whole installation fits on one node, and its maintenance is a break for everyone.
    • Students' code has no internet. pip install in a cell doesn't work; libraries come only in the catalog's kernel images, built outside the cluster.
    • No per-room disk quota: the colloq-workspace volume is shared.
    • Network policies are not all of the isolation. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node, and Pods share the node's Linux kernel.
    • A GPU submission's share is the device plugin's. Colloq neither caps nor watches video memory on Kubernetes: a MIG slice has memory of its own, while time-sliced replicas share a card's memory and compute with nothing between them, and one submission can take the memory the others need.
    • amd64 only.
    • Kernels on Debian 12. A scanner finds critical findings in them that Debian has not fixed; see “Images and the registry mirror”.
    • A new path. The chart is new in this release: test the installation in a non-production namespace before running classes in it.
    
 
    
 
    Search the docs
    Start typing a title or a question.
    Search runs in your browserEsc close
    
\ No newline at end of file
diff --git a/site/docs/en/search-index.json b/site/docs/en/search-index.json
index 6df9341e..3b36b429 100644
--- a/site/docs/en/search-index.json
+++ b/site/docs/en/search-index.json
@@ -1 +1 @@
-[{"title":"One link. The whole class.","section":"Get started","url":"./","text":"A guide to Colloq: from your first class to your own server."},{"title":"How a room works","section":"One link. The whole class.","url":"./#room","text":"A teacher creates a class on their own Colloq server. Students open its link and enter a name. In a standard class the group shares one notebook, one set of files and one Python kernel: a variable defined in the first cell is available in the next. A room can hold several notebooks, and they all run in the same kernel. The class workspace. The teacher decides which actions are available."},{"title":"Pick a format","section":"One link. The whole class.","url":"./#choose","text":"Hands-on class Write and run Python together → Lecture Show a PDF, write by hand, connect a tablet → Council Collect individual answers and discuss them with the group → After class Put up the class materials and build a course →"},{"title":"Before you start","section":"One link. The whole class.","url":"./#status","text":"Colloq runs on your own server. This site holds the documentation and the product overview; the class room itself opens at the address your teacher shares. Colloq is an open-source project under the MIT license. The source code is at github.com/colloq-edu/colloq , and server releases are published on its Releases page. To run a class on your own computer, install the package and start the server: pip install colloq colloq start You need Docker (on macOS 14+ and Linux x64/arm64 the package brings its own Node.js; elsewhere it needs Node.js 22 or newer); class data is stored in ~/.colloq . The server guides describe a versioned process with a manifest and images; they are no substitute for testing a specific release on your own machine. To work on the source code, use development mode . Each room gets its own kernel and file directory. Participants in the same room share one runtime. For details and limits, see the isolation guide ."},{"title":"Your first class","section":"Get started","url":"getting-started.html","text":"From signing in to the panel to the link your students open."},{"title":"1. Sign in as a teacher","section":"Your first class","url":"getting-started.html#access","text":"Open the personal sign-in link you got from the server owner. It looks like https://your-server/admin/k/… and opens the teacher panel. No password or email confirmation is needed. If you are setting up your own server for the first time, open the /admin/t/… link: until the server has an owner, it prints this link to the log on every start. You can also open /admin and paste the setup token by hand. The token is stored in DATA_DIR/setup-token : /var/lib/colloq/data/setup-token in a standard k3s install, ~/.colloq/data/setup-token after pip install colloq . Enter your name and email and click “Claim this instance”. The same token lets you recover access later. Two different links An /admin/k/… or /admin/t/… link gives admin access. Send your group only the class link, /s/… . Never show your personal key on the projector. Button names in this guide are given for the English interface. The owner can choose English or Russian for the whole server."},{"title":"2. Create a class","section":"Your first class","url":"getting-started.html#create","text":"In the panel, open Classes and click “New class”. Give the class a title, then choose a Python environment, a starting notebook (“Blank”, “From a file” or “From GitHub”, public repositories only) and a format. Under “Resources” you can set the memory and the number of processor cores for the room's kernel. You can also raise the memory during class from “Settings” in the class menu; the kernel does not restart, whether rooms run in Docker or on k3s. See Memory and CPU for a class . Format Starting rules When to choose it Standard The group edits the shared notebook and runs code. Practice and working through problems together. Lecture Editing, running code, notebook structure and files stay with the teacher, who opens individual cells to the group. Explaining with a live demo. Council Lecture rules; opening a cell gives everyone their own sheet. A task followed by a discussion of the answers. These are starting sets of rules , and you can change them during class. Oracle access (“As set for the instance”, “Off”, “Hints only” or “Full answers”) is chosen when you create the class, within the limits of the server's mode; it cannot be changed afterwards."},{"title":"3. Prepare the workspace","section":"Your first class","url":"getting-started.html#prepare","text":"Add code and notes to the notebook; upload a notebook and data if you need them. Run the setup cells. Make sure the libraries you need are available in the chosen environment. Check what students are allowed to do: editing, running code, files, the oracle and the shared screen. Open the class link in a separate private window and join as a student. Check this exact view before class starts. For a lecture, upload the PDF in advance and check the projector and console . For a Council, run the shared setup code before you open the task."},{"title":"4. Invite your group","section":"Your first class","url":"getting-started.html#invite","text":"Copy the class link from the panel and post it in your group chat. Students just open it in a browser and enter a name. There is no need to install Python on students' computers: code runs on the server. If the link contains localhost , fix the server's public address first. On your own computer, colloq start --share gives the class a public address and prints the link to send ( one link ). Check that the link opens from another network, not only from your own laptop."},{"title":"5. End the class","section":"Your first class","url":"getting-started.html#finish","text":"In the room, click the lock button, “What you can do in this room”, then End class ; in the panel, the same action is End the class in the class menu. Students can still reread the materials, files, terminal history and oracle answers, but only the teacher can act. Resume class in the room, or Reopen the class in the panel, brings the original rules back. If the room is in a course, Colloq then offers to choose materials for the class page : notebooks with their results, slides and data at one permanent link. Ending a class, publishing it and deleting the room are three different actions."},{"title":"Access for other teachers","section":"Your first class","url":"getting-started.html#team","text":"The owner adds teachers under Who can teach : click “Add a teacher”, enter a name and email, and confirm with “Add teacher”. The name and email only identify the person; Colloq sends no email. Hand the new sign-in link to the teacher privately. If a link is lost or has leaked, click “Rotate sign-in link” in that teacher's row: the old link and every session opened with it stop working."},{"title":"Owners and teachers","section":"Your first class","url":"getting-started.html#roles","text":"The panel has two roles. An owner sees and changes everything: every course, every room, the staff list and the server settings. A teacher sees only what they teach: their own courses and rooms. That is how courses from different departments share one server: nobody browses other people's classes or ends up running someone else's room. Owner Teacher Courses and Classes Everything. A Mine / All switch: Mine by default, which is what they teach themselves; All shows their colleagues' courses and rooms. The courses that list them as teaching, the rooms of those courses, and the rooms they host. Host in a room In any room. Only in the rooms they can see in the panel. Add a teacher To any course or room; on the Who can teach screen, the Courses column. To their own course or room, including an invitation by email. Delete a course or room Yes. No. Oracle, Environments, Resources View and change. View. Resources lists only their own rooms; the memory and CPU totals cover the whole server. Panel links to someone else's courses and rooms answer “not found”, as if they did not exist. The public /c/… and /p/… pages do not change: anyone with the link can still read them. Nothing disappears after the upgrade to 0.19 On the first start of the new version, everyone already on the staff list becomes a teacher of every existing course and of every room outside a course, so everyone sees exactly what they saw before. From then on an owner removes people from courses they do not teach, and new teachers start with an empty list: someone adds them to a course, or they create their own."},{"title":"Course teachers","section":"Your first class","url":"getting-started.html#course-teachers","text":"Whoever creates a course teaches it from the start. Under the course title is the Teaching strip with names; “you” marks yourself. “+ Add a teacher” searches the staff list by name or email. If the person is not there, “Invite by email” creates them and adds them to the course at once: the sign-in link is shown once, so hand it over privately, just like a link from the owner. If the email is already on the staff list, the person is simply added to the course and no new link is made: they already have one. A course's teachers host every room in the course: the ones they created, the ones a colleague created, and the ones an owner put into the course. Taking a room out of the course takes it away from the course's teachers, except those who host it personally. A room with no hosts of its own that sits in no other course stays with the course's teachers as its hosts — otherwise only the owners would see it. The same goes for deleting a course. Changes reach open rooms at once, with no server restart. You can always remove yourself from a course. Only an owner can remove a course's last teacher, so that no course is left with nobody but the owners."},{"title":"A class without a course","section":"Your first class","url":"getting-started.html#no-course","text":"A course is optional. On the New class screen the Course block has a “No course” card: the room stands on its own, and it is seen by whoever created it and by the people they add. They are added in the class settings, as the room's hosts. Only an owner can remove the last host of a room outside a course. If you pick a course, the room goes straight into the chosen plan row, taking its topic and day, or at the end of the course. A course room can have its own hosts too: for example, a teaching assistant who runs one class without teaching the whole course."},{"title":"You opened someone else's room","section":"Your first class","url":"getting-started.html#guest","text":"A teacher who opens the link to a room they do not teach comes in as an ordinary participant: under their own name, under the room's rules, without control of the kernel, the rules, the lecture or bans. A strip at the top of the room says “This room belongs to the course “…”. You are here as a participant.” and who can run it. To become a host, ask any of them to add you to the course or to the room's hosts, or ask an owner. Once you are added, reload the page: the room opens as yours. Entering from a class page works the same way: getting past the “Participants” and “No one” settings is only for the teachers of that room."},{"title":"For students","section":"Get started","url":"students.html","text":"How to join a room, take part in a class and come back to the materials."},{"title":"Joining with a link","section":"For students","url":"students.html#join","text":"Open the /s/… link your teacher gave you and enter your name. You don't need an account, an email address or Python on your computer. Stay in the same browser: part of your identity and your local drafts are tied to it. The /admin address is for teachers. colloq.ru/docs/ is documentation; it does not connect you to a class. If you only have the server address, ask your teacher for the link to your room."},{"title":"What's on the screen","section":"For students","url":"students.html#workspace","text":"Notebook — code, text, run results and other participants' cursors. Files — the room's data and documents. The file you have open may differ from the document on the shared screen. Oracle — questions to the AI and its answers, if it is enabled for the class. People — who is in the room right now; the panel is on the left, next to the files. Terminal — the room's shared shell: everyone sees its output, and you can type commands if the teacher allows running code. On a narrow screen the side areas open as panels. Close them with their button or Esc to get back to the notebook."},{"title":"Working in the shared notebook","section":"For students","url":"students.html#participate","text":"In a standard class, cell edits and output are visible to the whole group. Code runs in the shared kernel: changing a variable or a file affects everyone who runs code after you. Check with your teacher before you restart the kernel or clear output. If an action is unavailable, read the note next to it. The teacher may have kept it for themselves, locked the cell for now or ended the class. Reloading the page does not change these rules."},{"title":"Your own answer in a Council","section":"For students","url":"students.html#own-answer","text":"When the teacher opens a task in Council mode, edit your own sheet and send your answer with Submit . To change a submitted answer, click “Edit”. If the teacher allows running code, “Run” puts your attempt in the shared queue, either right away or after the teacher approves it; otherwise you will see “the teacher runs this one” next to it. The group does not see your answer until the teacher shows it. The teacher can reply to you and show a chosen solution to everyone, with your name or a variant number. Your own sheet does not mean your own Python kernel. Code runs in the runtime of the whole room, but each attempt gets personal copies of its data, so changing a table or a list does not reach the next person. Files on disk and anything the server could not copy stay shared — when that happens, your output says so by name. More in Council ."},{"title":"After class","section":"For students","url":"students.html#reading","text":"The link of an ended class still opens the saved work for reading. A /p/… link opens the class page: the materials your teacher put up — notebooks with their run results, slides, data. You read a notebook right on the page, can copy code from its cells and download each notebook as .ipynb with its results; you can't run it on that page. “Download all” gives you every material in one archive, at the same paths as in the room, so the code finds its data after unpacking. If you were in the class, the page has a “Class room” block with a “Go to the room” button: the edit history, the marks and the questions to the oracle are still there. Usually the button shows only in the browser you joined from. A course link, /c/… , is the timetable by day: what happened last time, what is next, and every class of the term with a link to its page. Bookmark it: materials usually appear there on the day of the class, after it ends. While a class is on, the course page also has an “Enter the room” button — if you have joined the room from this browser before."},{"title":"If the connection drops","section":"For students","url":"students.html#connection","text":"Wait for the page to reconnect, then check that your last text is there. The connection, output and code execution depend on the server being reachable; closing the tab does not stop Python. Don't send the same run over and over. If the server is unreachable, send your teacher the page address, the time and the error text, but no personal sign-in links."},{"title":"Room rules","section":"Run a class","url":"room-rules.html","text":"Control who edits, runs code and changes the shared work."},{"title":"The format sets the starting rules","section":"Room rules","url":"room-rules.html#presets","text":"“Standard”, “Lecture” and “Council” are sets of starting values. You set the rules in the new class form and change them later: in the panel, from “Settings” in the class menu, or in the room itself with the lock button, “What you can do in this room”. Changes apply at once and participants don't need to rejoin; the server checks the rules on every action. The teacher can open a single cell to the group and keep the rest of the notebook locked."},{"title":"What you can set","section":"Room rules","url":"room-rules.html#rules","text":"Setting What it means When I open a cell to students “They write together”: one shared cell for everyone. “Each answers alone”: every student gets an answer of their own, and the teacher sees them all in the console (Council). Type in cells Everyone or only the teacher. Run code Everyone, “One at a time” or the teacher. “One at a time” allows each student at most one cell per notebook, queued or running; “Run all” and running all cells above stay with the teacher. The rule also covers running files and the terminal. Dangerous commands Whether cells may run commands that shut the kernel down or wipe the whole room’s state: exit() , quit() , os._exit() , os.abort() , a fatal signal aimed at the kernel itself ( os.kill , signal.raise_signal ), %reset , %xdel , and among shell commands — kill , pkill and killall aimed at the kernel or at “everything”, shutdown , reboot , halt , init 0 and rm -rf of the class folder. Blocked by default: one such line in a shared notebook costs the whole class its variables. The rule applies to the teacher’s cells too — the kernel is shared, and there is a Restart button for that. sys.exit() is left alone: the kernel survives it. Inside a council attempt these commands never run, whatever this rule says. Add and remove cells Everyone, “Add only” or the teacher. With “Add only”, a student cannot delete even their own cell. Put a document on the shared screen Who controls the shared screen; everyone can still browse documents on their own. Upload and edit files Everyone or the teacher. Downloads are not affected by this rule; deleting, renaming and replacing an existing file by upload stay with the teacher. Students’ personal notebooks Whether a student may start their own notebook in the room — empty, or by bringing in an .ipynb from the class folder. An own notebook is always personal: its author and the teacher edit and run it, everyone else watches. Off by default. This is not the same as the row above, which is about the class’s shared folder. At most three own notebooks per person; the author may also remove their own. Oracle in Act mode For whom the oracle reads and changes files and runs code by itself: everyone, the teacher, or nobody. Actions per request The limit on oracle actions per request in “Act” mode: each read, edit and run counts separately. 0 means no limit, an empty field means the server setting; if the server sets a limit, a class can only lower it. Oracle questions per hour Limit per participant, no higher than the server's. Does not apply to the teacher. Time between questions No shorter than the server's interval. Does not apply to the teacher. View version history Who sees earlier versions of the notebook and who made each change. Restart the kernel Clears the notebook’s variables for everyone working in it; cell text and files are kept. In a personal notebook its author restarts the kernel. Clear all output The output of every cell in the notebook, the terminal history and the oracle feed. Clearing a single cell depends on permission to edit it."},{"title":"The cell lock","section":"Room rules","url":"room-rules.html#locks","text":"The teacher picks one of three states: Closed — the teacher edits and runs the cell; Open to all — the group types in this cell and runs it, even if the room rules keep that for the teacher, but can't delete or move it; Council — separate answers from each student. The “When I open a cell to students” setting decides what a click on the lock does; all three states are in the menu that opens when you press and hold the lock button or right-click it."},{"title":"Access to a single notebook","section":"Room rules","url":"room-rules.html#book-access","text":"The room rules apply to every notebook at once, but each notebook also has an access of its own. You change it from the “Access” menu on the notebook's tab; the menu is the teacher's. Access What it means As in the room The class rules, unchanged. The default for every notebook, including the room's own. Personal — author The author and the teacher type, rearrange and run; everyone else watches. The author is whoever started the notebook during the class; a notebook created by the teacher has no author, and this option is unavailable for it. Open to everyone Any participant may type, rearrange and run, even if the room rules keep that for the teacher. Teacher only Only the teacher types and runs, even if the room is open. Access overrides exactly three rules — “Type in cells”, “Run code” and “Add and remove cells” — and what follows from them: formatting, “Apply” on an oracle proposal, clearing your own cell's output, moving cells, “Run all” and running cells above in that notebook. Everything else stays shared by the room: files, the terminal, the shared screen, restarting the kernel, history and the oracle. Access belongs to the notebook, not to the file: renaming the file keeps it, removing the notebook from the room clears it. When access is not “as in the room”, the tab carries a short label: “personal · name”, “mine”, “open to everyone”, “teacher only”. The teacher grants the rights Students’ personal notebooks are off by default: “Students’ personal notebooks” is set to “not allowed”. Otherwise, in a lecture where the teacher types and runs, any participant could grant themselves those rights with one “new notebook” click. When they are allowed, a student's notebook becomes personal at once, with whoever started it recorded as its author. A personal notebook has a Python kernel of its own in a separate container: the class's variables do not reach it, it is never given the class's GPU, and its author may restart its kernel and clear its output. A notebook the teacher started has no author and stays “as in the room”; the teacher changes any notebook's access from the “Access” menu. “Own notebooks” and “Upload and edit files” are different rules. The notebook file is written by the server — it is a projection — so a student can start their own notebook even where the class files are the teacher's. And the other way round: you may drop an .ipynb into the folder, but bringing it into the room as a notebook needs own notebooks to be allowed. “End class” is stronger than notebook access: after the bell, personal and open notebooks are locked like everything else. “Resume class” brings them back."},{"title":"Action rules and Python execution","section":"Room rules","url":"room-rules.html#boundaries","text":"Mind how permissions combine If students can edit code, the teacher may unknowingly run text that someone has already changed: the source is read when the queue reaches the cell. For a strict lecture, restrict both editing and running. The “Dangerous commands” rule is a speed bump, not a sandbox. It stops the lines a student types out of curiosity; it does not stop intent — ctypes , reloading the os module through importlib , a fork bomb, a segfault in a native library and globals().clear() all walk past it. The room’s terminal is a real shell and the guard does not reach it at all: whoever the “Run code” rule lets into the terminal (by default the whole room in a standard class, the teacher alone in a lecture or a council) can drop the kernel from there with one command. Running Python has access to the room's files. Blocking uploads or edits in the interface does not stop code that is allowed to run from writing a file with open(...) . Interface rules do not give each participant a separate file system."},{"title":"Ending a class and moderation","section":"Room rules","url":"room-rules.html#moderation","text":"“End class” leaves all actions to the teacher for now and keeps the settings and materials. “Resume class” brings back the previous rules. An empty room only means nobody is connected; it does not mean the class has ended. To remove a participant, use “Remove from class…” in the “People” panel. Their access is blocked for 24 hours, tied to a marker in the browser; another browser or a private window can get around it. The participant's oracle questions and the answers to them are deleted, and version history won't bring them back; their Council attempts and run requests are deleted too. To lift the block early, click “Restore” in the “Removed from the class” list; deleted content is not restored. This is a tool against disruption, not identity verification or exam proctoring."},{"title":"Notebook, files and terminal","section":"Run a class","url":"notebook.html","text":"A shared Python workspace: what is shared, what is kept and how to run code."},{"title":"Cells and the shared document","section":"Notebook, files and terminal","url":"notebook.html#cells","text":"A notebook is made of Python code cells and text cells. Edits sync between participants. A cell's output appears below it: text, tables, plots or a Python error. What you can do depends on the room rules and the position of the cell's lock. Keys in the code editor Action Shift + Enter Run and move to the next cell. Ctrl / ⌘ + Enter Run the current cell. Alt + Enter Run and add a cell below. The run buttons also work on devices without a keyboard. To continue a notebook you prepared earlier, upload the .ipynb to the room's files and open it: it becomes the room's notebook. Code, text and embedded images carry over; saved outputs do not. You can download the notebook file from the Files panel, but it contains only the cells' code and text, without outputs. A room can have several notebooks: each one is an .ipynb file in the Files panel, and “New notebook” creates another. Every notebook has its own Python kernel: the lecture's variables do not appear in the seminar, and the two can be computing at the same time. A dot on a tab means that notebook's kernel is busy right now. Version history and restore cover only the room's main notebook, whose name is shown in the History panel; edits in other notebooks do not appear there."},{"title":"What appears under a cell","section":"Notebook, files and terminal","url":"notebook.html#outputs","text":"A cell can show: text and Python tracebacks, pandas tables (styled df.style included), Markdown from display(Markdown(...)) , images from matplotlib and other libraries, SVG, and interactive plotly figures. Large images and figures are not kept inside the shared document: they live next to the room and arrive in a separate request, so a heavy chart does not slow down everybody else’s typing. plotly works out of the box: it is in the base image, nothing has to be installed, and the plotting library is served from this server — no internet is needed in the classroom. As in Jupyter, only the last expression of a cell is shown; to show them all, add .show() to each: import plotly.express as px px.histogram(apartments, x=\"metro_time_min\", nbins=28).show() px.scatter(apartments, x=\"area_sqm\", y=\"price_rub\") A figure can be zoomed by dragging, hovered for values and filtered through its legend; the mouse wheel over it scrolls the notebook instead of zooming the chart. It is drawn in a sandbox of its own inside the page, with no access to the class and no network at all — so an image referenced from layout.images by URL will not load. A very large figure (over 12 MB of data — hundreds of thousands of points) is not shown: the cell gets a line suggesting fewer points, or saving the figure to a file with fig.write_html(\"plot.html\") . Outputs that need to run their own JavaScript — bokeh , folium , ipywidgets , altair without an image — are not executed: a cell’s output is produced by anyone allowed to run code, and it is rendered in every browser in the class. Instead of empty space, such a cell shows a line saying the output is interactive. Save it to an .html file, or produce an image."},{"title":"One kernel per notebook","section":"Notebook, files and terminal","url":"notebook.html#kernel","text":"All cells in ONE notebook share the same Python memory. Run the setup once, then use its values in later cells of that same notebook: # Teacher's setup numbers = [2, 4, 6] # A later cell in the same notebook sum(numbers) / len(numbers) The order in which cells ran matters more than where they sit in the notebook. If a variable is not defined, run the cell that creates it. Running a cell again may write data a second time or modify an object that has already been modified. Restarting the kernel wipes that notebook's variables for everyone working in it; the class's other notebooks are untouched. The notebook text and files are kept. After a restart, run the setup cells again. Reloading the page is not the same as restarting Python. The kernels of a class's notebooks live in one container and share its memory and CPU. Students' personal notebooks are the exception: their kernels run in a separate container with its own memory limit, and they are never given the class's GPU. A heavy cell in somebody's draft therefore cannot take down the lecture's kernel. A personal notebook's kernel that has idled for half an hour stops on its own — run a cell and it comes back."},{"title":"Working with files","section":"Notebook, files and terminal","url":"notebook.html#files","text":"The Files panel shows a folder tree. You can upload a file from your computer, open it and download it. The room rules decide who can create and edit files; only the teacher can rename them, move them to another folder or delete them. A single file can be up to 50 MB, and all files in a room together up to 1 GB; the server owner changes these limits with the MAX_UPLOAD_MB and MAX_SESSION_MB variables. Text files can be edited together. Python sees this room's working files. Use relative paths so the notebook stays portable: from pathlib import Path Path(\"results\").mkdir(exist_ok=True) Path(\"results/answer.txt\").write_text(\"42\\n\", encoding=\"utf-8\") A downloaded .ipynb is not a backup of every file on the server. Before working on your own, also download the data, images and anything else the code depends on."},{"title":"Shared terminal","section":"Notebook, files and terminal","url":"notebook.html#terminal","text":"The terminal runs commands in the room's environment and shows a shared output history. The “Run code” rule applies to it too. The command python analysis.py starts a separate Python process: notebook variables do not carry over to it, but the room's files are still shared. The “Run” button in the editor of a .py or .sh file saves your latest edits and runs the file in the same terminal. In production, the room's root file system is read-only, and the room has no network access by default except DNS. Installing packages from the internet during class may not work. Add the libraries in advance with an environment release ."},{"title":"What survives a stop","section":"Notebook, files and terminal","url":"notebook.html#persistence","text":"The notebook, files and saved materials are stored on disk. Python memory and processes live only as long as the kernel. A server update or a consistent backup stops the kernels, and their memory is lost. If a room has had nobody in it and nothing running for two hours, the server stops its kernel on its own: the variables are lost, the notebook and files stay. Deleting the server's disk destroys the files too, unless you have an off-server backup ."},{"title":"Oracle","section":"Run a class","url":"oracle.html","text":"Set up AI help, question limits and file editing by the oracle."},{"title":"Turning it on for the server and a class","section":"Oracle","url":"oracle.html#enable","text":"The owner sets the provider, base URL, model and API key on the Oracle page. Settings saved there can override the server's environment variables. The provider must offer an OpenAI-compatible API. Providers that require authentication need a key; Ollama and vLLM can work without one, as long as the server can reach the endpoint and the model name is correct. When you create a class, you choose its oracle access: As set for the instance (the default), Off , Hints only or Full answers . A class can only make the server's mode stricter: if the owner chose “Hints only”, the room gets no full answers. This setting cannot be changed later. If a test should run without AI, create a separate room with the oracle turned off."},{"title":"Questions and context","section":"Oracle","url":"oracle.html#ask","text":"Open the oracle panel and ask a specific question: which cell you are working on, what you expected and what you got. The request may include the room's code and context, so do not put API keys in the notebook, or any data that must not be sent to the chosen provider. Answers appear in the room's feed. Check the suggested code and explanations: the model can be wrong, and neither hints mode nor extra instructions guarantee how it behaves."},{"title":"Act mode","section":"Oracle","url":"oracle.html#agent","text":"When the Oracle edits files directly rule allows it, the oracle can read, create and change files and run code. The rule can grant this to everyone, to the teacher only (the default) or to nobody. In “Hints only” mode, Act is not available to anyone. A room runs one task at a time. The Actions per request rule caps the number of steps: reading, editing and running each count as a separate action. The server default is 24, and 0 means no limit; a class can only lower a finite server limit. The oracle cannot delete files or folders. In Act mode, permitted changes to files and cells are applied immediately, within the permissions of whoever made the request. In Ask mode, a suggested cell change is applied only after you press “Accept”. Check which mode is selected before you send a task. Undo has limits Undo restores the file versions that are still available. It does not bring back the earlier state of Python memory, and it does not reverse every effect of code that has already run. Review the changes before you carry on."},{"title":"Limits and costs","section":"Oracle","url":"oracle.html#limits","text":"The server sets how many questions each student may ask per hour (20 by default) and the time between questions (0 by default, at most 300 seconds). A question limit of zero turns the oracle off in every class, for the teacher too. On top of the personal limit, there is an hourly cap for the whole class: 30 personal limits, or half a personal limit for each connected person once more than 60 are connected. A participant who joined less than two minutes ago has to wait before asking the first question. A room can lower the number of questions and lengthen the pause, but it cannot loosen the server's limits. An empty value in the room rules means the server setting applies; limits for participants do not apply to the teacher. With a paid API, the costs go to the account that holds the provider key. With a local endpoint, whoever runs it provides the compute. Do not set OPEN_SEMINAR_CREATION=true on a public server unless you really need it: anonymous class creation through the API can let strangers spend your quota."},{"title":"Class Oracle","section":"Oracle","url":"oracle.html#council","text":"In the Council console, open the Class Oracle tab and ask in your own words, or press a quick question: “Class status”, “Who is stuck”, “Common mistakes”, “Review the solutions”. You can ask at any time — even before anybody has written a line: the model sees the task text and the cell above it. Answers pile up as a thread and only the teacher sees them. The request works in “Hints only” mode too, but it counts toward the class's hourly cap. The provider receives the task, the context, the class figures, a line per person and their solutions. Student names go to the provider when this Colloq has “Student names in model requests” enabled (panel → Oracle, on by default). Turn it off and labels S1…SN go instead, with the server putting names back into the finished answer. Names and other personal data written inside the solutions themselves are sent either way. Names are highlighted in the answer: press one and that person's work opens. This helps the teacher lead the discussion; it is not automatic grading. Next to the question field is the reasoning level : instant, normal, thorough. “Normal” adds nothing to the request, “instant” asks the model to answer briefly without thinking out loud, “thorough” lets it think longer. The default is set in the panel; anyone may lower the level, only the teacher may raise it. On models whose reasoning cannot be switched off, “instant” gives a short answer but not a faster one — a faster answer means a different model."},{"title":"If there is no answer","section":"Oracle","url":"oracle.html#errors","text":"Read the refusal message. Each cause needs a different fix: the personal limit or the class's hourly limit is used up, the pause between questions has not passed, the participant joined less than two minutes ago, the room has too many requests at once, another Act task is already running, the class has ended, the oracle is turned off, or the provider returned an error. The owner checks the key, base URL, model and whether the server can reach the API. Do not paste the key into the room's chat to troubleshoot."},{"title":"Council","section":"Run a class","url":"council.html","text":"Everyone writes their own answer; the teacher chooses what to discuss together."},{"title":"Prepare the task","section":"Council","url":"council.html#prepare","text":"Create a class in the “Council” format, or pick “Council” from the lock menu on a single cell. The lock appears when the “Edit cells” or “Run code” rule is reserved for the teacher, as in the “Lecture” format. In a room where everyone can both edit and run code, there is no lock. In shared cells, prepare the data, the imports and the task statement. Run the setup code so the values you need are in the kernel. Open the task cell: each participant gets their own sheet. The Council format differs from Lecture in what opening a cell does. All other permissions can be set separately."},{"title":"Students answer separately","section":"Council","url":"council.html#submit","text":"Answers are stored on the server, separately from the shared document. A student writes their own version (up to 8,000 characters) and presses “Submit” or ⌘ / Ctrl + Shift + Enter . “Edit” takes a submitted answer back into work, and “Restore” resets the sheet to the original task. Until the teacher shows an answer, each participant sees only their own work, while the teacher sees both submitted answers and drafts. If an attempt fails with an error, the student can press “Ask the oracle”: only they and the teacher see the reply, and the question counts toward the oracle limit. Who runs attempts is set in the console's rules : Only me (the default), Everyone in turn or On request . The room's “Run code” rule does not by itself let anyone run Council attempts. In every mode, attempts run one after another in the room's single kernel."},{"title":"Council rules","section":"Council","url":"council.html#rules","text":"Council rules are set for each cell in the console window. Its header sums them up in one line, for example “Runs by me only · each run capped at 30 s · on screen with names”. To change a rule, press Rules , any value in that line, or the G key. Changes take effect at once; there is no Save button. Rule Options What it means Who runs code “Only me” (default), “Everyone in turn”, “On request” Students write the code and you run it; a student runs their own attempt; a student asks and you approve that exact version of the code. Changing the mode drops pending requests. Run limit 5 s, 15 s, 30 s (default), 1 min, 5 min, “No limit” If an attempt runs longer, the server interrupts it and the queue moves on. The limit applies to your own runs too. A new limit takes effect at once for waiting attempts and for the one that is running, counted from when it started. Below the options you can see how long runs of this cell usually take. Rerun pause “Right away” (default), 15 s, 30 s, 1 min, 2 min How long a student waits after their run finishes before they can run again or ask for a run. The pause does not apply to the teacher. If only the teacher runs code, this rule has no effect. On the class screen “With names” (default), “Without names” Without names, a shown solution is labeled with a number, such as “Variant 12”, the same way in everyone's notebook and on the projector. When the limit stops a run, its output shows “Stopped: the run took longer than …”, and the console marks it “Stopped: longer than …”. An interrupt cannot stop every kind of code. If an attempt does not stop, the console says “not stopping — interrupt it or restart the kernel”. During the pause, the student sees a countdown, “Run in …”, instead of the button."},{"title":"Running on request","section":"Council","url":"council.html#run-requests","text":"The student presses Run (or Shift + Enter ) on their sheet; in this mode, that sends a request to the teacher. While the request waits, the button is replaced by “In queue” with a “cancel” link; if it is declined, the student sees “The teacher did not run it. You can press again.” A student can run or request a run only before submitting: a submitted answer has no run button, and it comes back after “Edit”. Editing the text withdraws the earlier request, so the new version needs a new one. In the console window, requests are collected on the Queue tab under “Waiting for your approval”. The link “N awaiting approval →” appears next to the tabs, and in the work list such work is marked “Run requested”. Check the code and press Approve or Decline ; Approve all approves every pending request at once. Approval puts the version you checked into the shared queue; no code runs before approval. A request does not count as submitting the answer. Changing who runs code or closing the Council drops the requests."},{"title":"Discuss the answers with the group","section":"Council","url":"council.html#discuss","text":"You review work in a separate console window. Under the Council cell, the teacher sees only a counter (“N submitted out of M”, drafts) and the Console button. Do not put this window on the projector: it shows names, drafts and marks. If the browser did not open the window, allow pop-ups for this page. On the Work tab, the list of submissions is on the left: newest first, with three or more identical answers folded into a group. The selected work opens on the right. You can run it, mark it Correct or Revise , write to its author, and press Show the class . The code then appears under the cell for everyone as a labeled block, and the cell's own text does not change. Clear the screen takes it down. The author sees your mark (“✓ Correct” or “✗ Has an error”) and your messages under their sheet. The “Class Oracle” tab lets you ask the oracle about the class — the task, who is stuck, the solutions themselves — if the oracle is set up. Keys in the console window: J / K or ↓ / ↑ move to the next and previous work, Space expands a group, ⌘ / Ctrl + F searches by name, Enter shows the work to the class, R runs it, 1 marks it “Correct”, 2 marks it “Revise”, 3 clears the screen, ← / → move to the neighboring work in the same group, G opens the rules, and ? lists all keys."},{"title":"What stays shared","section":"Council","url":"council.html#shared","text":"Personal answer, the notebook's kernel, personal data Attempts run one at a time in the kernel of the notebook holding the council cell: every notebook has its own. An attempt sees the df , np and other values the teacher prepared in that notebook — but it works on personal copies of them. When it finishes, the server restores the namespace exactly as it was: it removes the names the attempt created and undoes its rebindings. So df = df.dropna() or df.drop(…, inplace=True) in one attempt does not reach the next. It is still not an isolated exam run. Personal copies are made of pandas frames and series, numpy arrays, lists, dicts, sets and tuples, PyTorch tensors and models together with their optimisers, SciPy sparse matrices, scikit-learn estimators and objects of classes defined in the notebook itself. The same entry also closes the three ways one attempt could end the class for everybody: exit() and quit() are refused (in a Jupyter kernel they shut the process down and everyone loses their variables), os._exit() and os.abort() are refused too (they end the kernel process on the spot, with no traceback at all, and the whole room's variables go with it), and an attempt's memory is capped at the container's limit, so np.ones((40000, 40000)) fails with MemoryError on its own card instead of taking the notebook's kernel with it. These refusals are a speed bump, not a sandbox: ctypes , signal.raise_signal and reloading os through importlib still reach the process. A child process after fork() keeps the real os._exit , so multiprocessing and joblib work as before. Process state goes back too: random seeds, sys.stdout , sys.path , os.environ , builtins, warning filters, numpy and pandas print options, matplotlib figures. Files on disk, modules the attempt imported, objects of types that cannot be copied (open files, generators, connections) and objects above the copy budget stay shared. Threads an attempt leaves running cannot be stopped — it is told how many in its own output. An attempt is told about the last of these in its own output, in a line like “Variable big (1.2 GB) is too large to copy for this attempt — it works on the shared one”. When that happens, work on a copy explicitly, as before: # df was prepared by the teacher local_df = df.copy() local_df = local_df.dropna() print(local_df.shape) Personal copies keep a class running, but they do not create a technical security boundary. Independent grading, tamper protection and identity checks need a separate system."},{"title":"A format that works in class","section":"Council","url":"council.html#practice","text":"Start with a short question, give time for individual answers, pick two different approaches and show them one after the other. Then bring the explanation back into the shared notebook and save the final correct version. Publishing the notebook does not export everyone's personal answers."},{"title":"Competitions","section":"Run a class","url":"competitions.html","text":"A task with a closed evaluation: data, a metric, a leaderboard and a submission queue."},{"title":"What this is","section":"Competitions","url":"competitions.html#what","text":"A competition is a task with a closed evaluation. You hand the class the data and the brief, an entrant submits an .ipynb notebook as a file, the server executes it from scratch and scores the result with your metric. The answers the score is computed against are never shown to an entrant; their place on the leaderboard is. A competition does not live in a room: it has its own address, /k/ , its own queue for the whole instance and its own entrants. No class is needed — a link is enough. You create and run competitions from the admin panel, under Competitions . This is not an exam system A submission runs in a throwaway container with no network, and its data is mounted read-only. But a competition does not check who is at the keyboard, and it does not stop two entrants from submitting one solution under two names. If a grade depends on the result, collusion and stand-ins are not something the server will catch for you."},{"title":"Set a competition up","section":"Competitions","url":"competitions.html#setup","text":"Open Competitions in the panel and press NEW COMPETITION . Give it a title and an address — the address becomes the /k/ link the class opens. Write the one-line blurb (it shows in the list) and the description in Markdown: the task, what is in the data, what to submit. Add the open data files . Entrants download them from the task page and see them inside the container as a read-only data/ directory. The ceiling is 200 MB per competition. Add the hidden answers through their own button. On disk the file is always called solution.csv and it lives outside the class workspaces, so no room can read it. The row key must be named id : it is what the rows are split by and what a submitted file is checked against. No other column name can be set yet, and a competition using one will not open — the baseline check catches it. If you need one, add a hidden test — input data the notebook reads during the check in place of the open example (see the hidden test ). The answers in solution.csv then belong to the hidden test, not to the example. Paste the metric code , or take a preset (MAPE, RMSE, MAE, ROC AUC, F1, accuracy, QWK) — a preset assembles itself from the columns of your answers file. Say which direction is better: lower or higher. Upload the sample notebook — the baseline every entrant starts from — and press Run the full check . Set a deadline and press OPEN COMPETITION . Until every step is done, the OPEN button stays disabled and names what is missing. The sample notebook is a gate of its own: until it has gone all the way to a number, the competition cannot be opened at all. A task that does not even solve for its author means a hundred people hunting for a mistake of their own that isn't there."},{"title":"Who sees a competition","section":"Competitions","url":"competitions.html#access","text":"An owner sees and runs every competition. A teacher sees the ones they created and the ones in their courses: the Course field in the editor hands a competition to that course's teachers. Anyone else's competition answers “not found” in the panel. The entrants' /k/… page does not change. After the update to 0.19 Competitions made before the update stay with everyone who was on the staff list at the time, as before. Once a competition is given a course, only that course's teachers, its creator and the owners see it. Teachers added later see only their own competitions and their courses'."},{"title":"The metric code","section":"Competitions","url":"competitions.html#metric","text":"A metric is one function. The server calls it with two pandas frames — your answers file and the submitted submission.csv , already joined on the id column. It must return a number. Before the metric, the platform checks the file itself: it reads as a table, has the id column and no stray index column ( Unnamed: 0 from to_csv without index=False ), repeats no id , misses none of your answer rows, leaves no prediction empty where your answers have a value, and where your answers hold numbers the predictions are numbers too, without inf — a comma used as the decimal separator is named as such. Ids are compared as the file writes them, as text: when they differ only in case, hyphens or spaces, the entrant sees one of their ids next to the test's. For a key of plain whole numbers, 7.0 and 007 in an answer count as id 7. A submission that fails is refused — the entrant reads what exactly is wrong — and the metric does not run. import numpy as np import pandas as pd class ParticipantVisibleError(Exception): pass def score(solution: pd.DataFrame, submission: pd.DataFrame) -> float: if 'orders' not in submission.columns: raise ParticipantVisibleError('The submission has no orders column') merged = solution.merge(submission, on='id', how='left', suffixes=('_true', '_pred')) if merged['orders_pred'].isna().any(): raise ParticipantVisibleError('Some rows of test.csv have no prediction') true = merged['orders_true'].to_numpy(dtype=float) pred = merged['orders_pred'].to_numpy(dtype=float) return float(np.mean(np.abs((true - pred) / np.maximum(true, 1e-9)))) The text of a ParticipantVisibleError reaches the entrant verbatim — that is how you tell them what is wrong with their file. Any other error is yours alone: the submission is marked metric failed , the entrant reads \"the checking code failed\", and you get the traceback. After fixing the code press Fix the metric and rescore everyone : notebooks are not re-executed, only the metric runs again. Check against the baseline runs the metric over the sample notebook's saved answer without executing it again — that turns a code fix into a few seconds of waiting."},{"title":"The public and the private share","section":"Competitions","url":"competitions.html#split","text":"The metric is computed twice: on the public share of the answer rows and on the private one. The public leaderboard is always visible — it is how entrants tell whether they are getting anywhere. The final one stays closed until the deadline: a hidden share you cannot fit to by refreshing the page is the whole point of having one. The public share is a number in the editor (30 % by default). Rows are split once and for good: the split seed is recorded when the competition is created, so every entrant and every submission faces the same public share. If your answers file has a Usage column holding Public and Private , the split comes from it and the share is not asked for. What counts is the submission the entrant picked with \"Count this one\". If they picked none, their best public one counts. The rule can be changed to \"best public\" or \"latest\" in the competition's settings. Places are never shared: every leaderboard row has its own, and on equal results the submission sent earlier ranks higher."},{"title":"The hidden test","section":"Competitions","url":"competitions.html#hidden-test","text":"An open data/test.csv is downloaded by the class, and an answer is easy to fit to it: count the rows, copy the ids. A hidden test is a file the entrant does not download: at every check the notebook reads it at the same path, in place of the open example. The model is checked on rows nobody has seen. In the editor's Hidden test section, upload the file through its own button — not through the open data. In Path during the check give its name in data/ ; by default it is the file's own name. A file named like an open example replaces the example; a file with a new name is added to the data. The example stays with the entrants: they download it and debug their notebook on it. Check the columns — the editor shows whether they match the example. A notebook written for the example fails on a test with other columns. The answers in solution.csv must belong to the hidden test's rows. Run the sample notebook's check again: a change to the hidden test makes its last check stale. Every run gets a data/ directory of its own: the open files, with the hidden ones in their place. data/test.csv and /data/test.csv both read the hidden test; the metric never sees it. A file replaced in the middle of a run reaches the next runs only. The ceiling is 200 MB and 10 files, apart from the open data. Entrants see the hidden file's name and row count: the example carries an EXAMPLE mark in the data list, and the check conditions have a line about the hidden test. No link hands out its contents — not to the entrant and not to you: the hidden test cannot be downloaded. While the notebook runs, the hidden rows are in its memory, and any output can carry them out: test.head() , a KeyError message with a value in it, a metric message quoting the answer. That is why What participants see of a check has two options: Only the outcome and the error type — the default. The entrant sees the outcome, the number of the failed cell and the exception type from a short list of common ones ( KeyError , ValueError …). No cell output, no error text, no traceback; the notebook link hands out the notebook as it was sent, not the executed one. A metric refusal is retold in our own words, without values, ids or counts from the answer. Full output — as without a hidden test. Turn it on only if revealing those rows does no harm: fitting the answer to them becomes possible. In the feed you always see the whole output and the executed notebook. The full text is kept under either option, so a switch applies to every submission at once, with nothing run again."},{"title":"How a submission is executed","section":"Competitions","url":"competitions.html#run","text":"The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in. Entrants download the open files one by one, or with Download all above the list: the archive holds data/ with every file and the starter notebooks next to it, as in the check, so pd.read_csv('data/train.csv') works right after unpacking. Hidden answers and the hidden test are never in it. The submission joins the instance queue. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv . That file is taken out of the container and the container is destroyed. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here. A submission gets no GPU A competition's environment is picked from the same list a class uses, but the submission container starts without a GPU , even when an environment with a card is selected. The libraries in the image are the same; the CPU does the work. A task that needs a card cannot be run as a competition yet. The competition's limits (notebook time, memory, cores) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full. Each submission has a \"Download the notebook with its output\" link: the notebook as the server executed it — a failed one too, up to the cell where it stopped. Its output is this run's alone: whatever came from the laptop is cleared before the run. Outcome What the entrant sees Done The public score and their place. Notebook failed \"Failed on cell 5 of 14\" and their traceback verbatim. Time limit \"Did not fit into 10 minutes, cell 6 of 50\". Out of memory \"Took more than 4 GB, cell 2\". Answer not accepted \"Reached the end but left no submission.csv \", \"The answer is missing 3 rows, for example id=17\" — or the text of your ParticipantVisibleError . Metric failed \"The checking code failed\", with no detail; you get the traceback and the submission waits to be rescored. That is a competition without a hidden test. With one, an entrant by default sees a failed submission briefly: the outcome, the cell number and the exception type, with no output and no traceback; see the hidden test ."},{"title":"The queue and the console","section":"Competitions","url":"competitions.html#queue","text":"There is one queue for the whole instance; it lives in the database and survives a server restart. How many submissions run at once is worked out from the machine by default ( Auto ): its cores and memory, less 2 cores and 2 GB kept for the class, divided by a new submission's default cores and memory. The owner can set a fixed number in the panel's Resources tab; see sizing . The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session. The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue , and a run in progress has a Kill button. To stop anyone from fitting the public leaderboard with dozens of submissions, set a daily submission quota . Only submissions that get a score spend it. Failed ones — a notebook error, the time or memory limit, a rejected answer, a failure of the metric or of the server itself — and cancelled ones do not; a queued submission holds a place for now and gives it back if it fails. A separate ceiling on failed submissions stops endless failures: at most 10 a day per entrant, or three times the quota if that is more (metric failures do not count toward it). An entrant who reaches it gets a refusal of its own, apart from the quota. With no quota (0) there is no ceiling either."},{"title":"Entrants and the sign-in key","section":"Competitions","url":"competitions.html#entrants","text":"An entrant is not a room member: their identity belongs to the whole instance. On joining, a person gives their Telegram username or email — the name they appear under on the leaderboard — and receives a sign-in key shaped K7Q-M2X-9FD and a link carrying it. The same key brings them back to their submissions from another device, another browser, or after clearing site data. A username or an address is one person: joining under one the competition already has is refused with a pointer to the sign-in key, and no second identity with another key is created. The key sits in the \"YOUR SIGN-IN KEY\" card on the competitions page — shown every time, not once. The key itself is not in the database: what is stored is its fingerprint and a copy encrypted with the instance secret, so a stolen database without that secret opens no one's account. If an entrant loses their key, issue a new one from the Participants tab. The old one stops working that same second, together with every tab still open on it."},{"title":"The deadline and the debrief","section":"Competitions","url":"competitions.html#finish","text":"When the deadline passes, submissions close by themselves and the final leaderboard opens — if the settings say \"open it by itself after the deadline\". The other option is I will open it by hand — at the review : then a button reveals the results and you choose the minute of the class when they appear. Finish now closes submissions early. After the deadline there is no need to press it: the competition counts as finished anyway — participants and the panel list it among the finished ones, and the console header offers Close late intake or Take late submissions instead of Finish now, plus Open the private leaderboard when the results open by hand. For those finishing their solution after the bell there is the Late submissions switch in Dates and standings. With it on, submissions keep being accepted after the deadline and after Finish now: they run, get a public score and spend the daily quota by the same rules as ordinary ones. But a late submission never counts: it is in no places, no leaderboard, no summary and not in the wait for the final results, and it cannot be picked to count. The entrant sees its final score when the results open. On their page it is marked “after the deadline · not counted”, and in the console it carries a LATE mark and a Late filter of its own. A submission is late when its upload began after the deadline; one begun on time and finished within two minutes stays in the standings. A late submission never takes the place of an on-time one still waiting for its check. While late intake is open, the lists say so: participants see the competition among the finished ones marked “late submissions open”, with a button, and the panel writes “late ones accepted” under FINISHED, with how many came. If you move the deadline later on a running competition, submissions sent before the new deadline become ordinary ones: they join the leaderboard and the standings. Moving it earlier makes no one late. After Finish now moving it changes nothing: the standings were closed by hand. For the debrief there is Leaderboard on the projector — a separate window with just the table, without file names or internal marks."},{"title":"Additional packages","section":"Competitions","url":"competitions.html#packages","text":"The Packages tab shows the pinned Python version and complete base inventory. Entrants can enter additional libraries one per line or import requirements.txt . Comments, on a line of their own or after a requirement ( numpy==1.26 # for the metric ), and blank lines do not count towards the 32-requirement limit; the whole request is limited to 8 KiB. The server checks compatibility, downloads PyPI wheels and verifies installation without network access. Base package versions cannot be replaced. Choose a ready set before explicitly confirming notebook submission. Its packages and base image remain bound to that submission; changing the current selection does not alter a rerun. The set is installed into the submission's memory at every run and counts against its limit: it may take at most half, the rest being the kernel's, the data's and the working folder's. A bigger set cannot be chosen, and the entrant sees how much it would take. On the Docker path, preparation requires Docker Engine 28 or newer. Notebook execution and scoring have no internet access."},{"title":"Checks and reruns","section":"Competitions","url":"competitions.html#revisions","text":"Every attempt has its own identifier and directories. Rescoring an active submission waits for that attempt to finish; an old CSV cannot become the result of a failed new run. Changes to the metric, data, base environment or starter notebook require an up-to-date baseline check. After changing notebook data or its environment, metric-only rescoring is insufficient: rerun the notebook."},{"title":"Executor availability","section":"Competitions","url":"competitions.html#availability","text":"Submissions and package preparation run where the rooms' kernels run: on the Docker path (the server image, colloq start ) in throwaway containers on the same Docker, on k3s in Pods the broker starts. If the executor or package preparation is unavailable, the interface says why and refuses the work before it enters the queue; /api/health reports each capability separately."},{"title":"Lecture, projection and console","section":"Run a class","url":"lectures.html","text":"Show a PDF and handwritten annotations on the projector and in students' browsers."},{"title":"Prepare the lecture","section":"Lecture, projection and console","url":"lectures.html#start","text":"Create a class in the “Lecture” format: by default, the teacher controls the notebook, code runs, files and the shared screen. Upload the slides as a PDF to the room's files, open it and press Lecture in the viewer. Anyone the “Put a document on the shared screen” rule allows onto the shared screen can start a lecture; only the teacher can end someone else's lecture. A notebook with examples can stay open alongside. The format does not rule out practice: the teacher can open a single cell to the whole group or switch on Council."},{"title":"Connect the projector","section":"Lecture, projection and console","url":"lectures.html#screen","text":"The Project button on the lecture bar opens the projection of this same room, /s/ID/screen , in a separate window: drag it to the projector screen, and the room stays available on your laptop. It is not a new room, just another view of the class. You can also open the projection address on another computer by signing in to the same room. Paging with ← / → , PageUp / PageDown or a presentation clicker at the projector works only if the projection is opened as the presenter; otherwise, page from the laptop or tablet. B or the “Pause” button blanks the projection, while the presenter's page stays. The projection is made for a big screen. It shows only the lecture page with its annotations, or a sheet to write on. Until the lecture starts, the screen says: “The screen is ready. It will show the document when the teacher starts a lecture.” The notebook is never shown on the projection: to show a cell, show the room window to the audience. Check the browser zoom, full-screen mode (the F key on the projection) and the device's connection to the server beforehand. When the presenter zooms in with the Magnifier , the projection shows exactly the same region as the console. The slide's edges on the screen stay where they are: the zoom moves smoothly inside them, and a moment after it stops the picture turns sharp. The projection is hard to close by accident. F5 , which the “slideshow” button of a presentation clicker sends, switches to full screen instead of reloading the page. To leave the projection, press Esc twice in a row: after the first press the bottom of the screen says “Esc again to close the projection”. Reloading the projection window during a lecture asks for confirmation first."},{"title":"Connect a tablet","section":"Lecture, projection and console","url":"lectures.html#tablet","text":"During the lecture, press Console on the lecture bar (tooltip: “Open console on a tablet”) and open the link you get on the tablet. The link is valid for 10 minutes and works once. It signs in to the room as you, so do not send it to students. The room's regular console address is /s/ID/pult , but on its own it does not grant teacher rights. The console has a pen, highlighter, eraser, pointer and magnifier , and you can add a sheet after any slide for a formula or a diagram. The group sees your handwritten annotations, while speaker notes are seen by teachers only. A second tap on Pen opens its palette. It has eight colours: black, red, blue, green, orange, purple, yellow and white, the last two for dark slides. With a keyboard on the tablet, keys 1 – 8 pick the colour. The five widths in the palette are drawn at their real size, exactly as the line will land on the slide. A second tap on Highlighter opens the highlighter palette: five colours (yellow, green, blue, pink and orange) and three widths. The highlighter lies under the ink, so the slide's text and your pen lines do not fade under it: black stays black. An underline, an axis or a fraction bar is easy to draw straight: hold the pen or highlighter still at the end of the stroke for about half a second, and the stroke straightens into a segment from its start to where the pen rests. Until you lift the pen you can still move the end, and a single undo removes the whole line. An arc, a bracket or a letter does not straighten when the pen rests. The pointer is red, and the projector and the students see it too. A second tap on Pointer offers two kinds: Line, to trace and write, and Dot, to point at a spot without a trace. A word written with the pointer stays lit while you keep writing and goes out as a whole 1.5 seconds after your last touch: all its strokes fade together over 0.6 seconds. The Notes key on the console rail opens the notes for the current slide. In landscape they stand as a column on the right; the slide gets smaller and stays available to the pen. In portrait the notes lie under the slide, and the border between them can be dragged with a finger or moved with the arrow keys; a double tap puts it back. Under the slide you see the page number, the lecture time, “planned” if the slides have timings, and a strip of slides and sheets that marks what comes next. Prompter (the T key, a button in the notes panel or a row in the “⋯” menu) hides the slide and the pen and leaves only the text in large type, 22 to 40 px. A thumbnail of the slide, the time spent on it and what comes next stay alongside. The “Slide” key brings the usual view back. The console remembers the type size and the chosen view on this device."},{"title":"Zoom in on a detail","section":"Lecture, projection and console","url":"lectures.html#zoom","text":"To show a small detail or look closer at a picture, spread two fingers on the console's sheet: the slide zooms in around the point between them, and moving two fingers together pans the zoomed slide. The other way is the Magnifier key on the rail: press it and draw a frame with the pen. The frame fills the sheet and the previous tool comes back, so you can write on the zoomed detail at once. A tap without a frame brings the slide twice as close around that point. A finger can draw the frame only until the console notices the pen: at that point the console says “Finger drawing is off: pen in use”, and from then on only the pen draws the frame on this device. The “Draw with your finger” row in the “⋯” menu turns finger drawing back on. You can zoom in up to ×4. With a keyboard on the tablet, + and - zoom in and out, and 0 shows the whole slide. While the slide is zoomed, the Magnifier key shows how close, for example “×2.5”, with “Whole slide” under it: a tap brings back the whole slide. Turning the page also opens the next page whole. The projector and the students who follow the lecture see the same region as you do. Annotations zoom with the slide, and a new stroke under the magnifier looks as thick on screen as it does without it, so on the whole slide it is finer, in scale with the detail. Pinching is for fingers only: the pen takes no part in it, and a palm resting on the glass is not a gesture. Put both fingers down together: if the second one lands noticeably later than the first, or the pen is near the glass, or a palm rests on it, the slide does not zoom. A short two-finger tap does not zoom; it undoes the last stroke."},{"title":"Sheets between slides","section":"Lecture, projection and console","url":"lectures.html#sheets","text":"When the slide is over but the derivation is not, press + Sheet on the console rail: a blank sheet goes right after the current page, whether that is a slide or another sheet, and the projection moves to it. The ‹ and › Next keys walk slides and sheets as one chain. The arrow keys and a presentation clicker at the projector or the laptop walk the same chain but never start a new sheet. A lecture can have up to 50 sheets; at 50 the console asks you to delete one you no longer need instead of adding another. The strip under the console's sheet shows slides and sheets in order and counts them: “Strip · 50 slides · 3 sheets”. In portrait the strip runs above the bottom rail. Tapping a thumbnail takes the lecture to that page, and the dashed “+” tile right after the current page does the same as + Sheet. Tapping the page number opens Pages , the same chain as a large grid. On a sheet, the page number reads “S2” with the caption “sheet 2 of 3 · after slide 14”. Sheets have no notes. Tapping the current sheet in the strip or in Pages opens the sheet's menu. Under Paper, pick Plain, Grid, Lined or Dots, and the hall sees the sheet on the same paper. Delete sheet asks for confirmation first and says how many strokes go with the sheet. If the hall was looking at that sheet, the projection moves to the previous page in the chain."},{"title":"Speaker notes","section":"Lecture, projection and console","url":"lectures.html#notes","text":"Notes are written for each page of a PDF and saved with the document. Only the room's teachers see them. The hall, the students and the projection never receive them, and they do not go onto the class page . Open the document and press Notes in the viewer bar. No lecture has to be running, so you can prepare the evening before. The editor opens over the room. On the left is the list of slides with the first line of each note and a “no notes” filter. In the middle are the current slide, a time estimate and the text field. On the right, As on the console shows the note as it will look on the tablet at 18, 22 or 28 px, and whether it fits one screen. On a narrow screen this preview is a tab next to the text. The text saves by itself a moment after you pause. The bottom line shows “Saved · HH:MM” and a counter up to 3,000 characters. Offline, the editor says “Offline — will save when it is back”. A note longer than 3,000 characters is not saved, and it is not cut either: the text stays in the field and the counter stays red until you shorten it, and closing the editor with such a text takes a second press. ⌘ / Ctrl + ↑ / ↓ opens the neighbouring slide, and Esc closes the editor. During a lecture the Notes button is on the lecture bar too, and a presentation clicker's PageUp / PageDown keep turning slides even while the caret is in a note. A note is Markdown, and the buttons above the field insert the markup. Bold marks the line the slide exists for. A line in italics is a cue for yourself and is dimmed on the console. Lists, quotes and formulas in $…$ work too. A line like *≈ 1.2 min* sets the time for the slide: the console shows it in the note's header and counts “planned” from it. A paragraph that starts with **If asked:** is an answer in reserve: on the console it is folded to the question and opens with a tap. Links and images show as plain text on the console, so a stray tap cannot take you out of the lecture. A whole script can be loaded at once. Import script (.md) takes a file up to 1 MB, a drop or text from the clipboard. Each section with a ## 14 · Title heading becomes the note for page 14. Headings like “## Slide 14”, “## Слайд 14: …” and “## 14) …” work too; sections without a number follow in order. Before anything is written you see which sections are new, which replace a note, which match it, and which will be skipped: over 3,000 characters, no such slide, empty or repeated. An empty section never deletes a note. Everything is written in one go, or nothing is. Download .md saves the notes in the same format, with slide titles from the PDF, and such a file loads back unchanged. Notes are tied to the file name and the page number: if you replace the PDF with one where slides moved, the notes stay on the old page numbers."},{"title":"Taking over a lecture","section":"Lecture, projection and console","url":"lectures.html#takeover","text":"One person presents a lecture: whoever started it. The same teacher in a second browser and a second teacher in the room see above the slide who is presenting and from where, for example “Ada is presenting — from another device (iPad, console)”, together with whether the presenter is online, when they last did something and which page they are on. In the people list your second browser reads “You · another device”. Watch as the hall folds this bar, and the Notes button next to it opens the notes for the slide. Take over the lecture moves control here. The page, the sheets, the zoom, the handwritten annotations and the lecture clock stay, and the line the previous presenter was drawing stays with the hall. If the hall was looking at a zoomed slide, the presenter's bar on the laptop shows a key with the zoom, for example “×2.5”: it gives the hall the whole slide back, and so does the 0 key. If a colleague is presenting and is online, press the button a second time. Your own lecture on another device, or a presenter who has been offline for more than 5 seconds (a closed laptop, a dead tablet), is taken over with one press and no waiting. The previous presenter sees “The lecture was taken over” and can Take it back or Watch . On a tablet console that is not presenting, the turn keys are replaced by who is presenting, and the “Take control” key (“Present from here” when it is you) takes the console. A take-over needs a connection to the server: offline the button is disabled, and a press is never queued until the connection returns. If the lecture has ended by then, or someone else has taken it over in the meantime, the server refuses and says why. The Lecture button on a document another teacher is already presenting does not take the lecture over; it points you to Take over the lecture. Only teachers can take over a lecture."},{"title":"How students follow the lecture","section":"Lecture, projection and console","url":"lectures.html#follow","text":"Students see the slides and annotations in their own browser. While a student follows the lecture, they see what the projector shows: the same page or sheet, on the same paper, at the same zoom. Read on my own lets them go: they can leave the current page, go back to an earlier one and finish reading, and the presenter's magnifier does not move their view. To lecture brings them back to the presenter's page. The “Put a document on the shared screen” rule controls the shared screen; it does not stop anyone from viewing documents on their own."},{"title":"Materials after the lecture","section":"Lecture, projection and console","url":"lectures.html#after","text":"When the class ends, the room stays open for reading. Handwritten annotations, sheets with their paper and the zoom live only in server memory: the “End” button on the lecture bar, a server restart or a server update erases them for good. A server restart during a class also ends the lecture itself, and it has to be started again, so update the server between classes. Speaker notes are saved page by page for each document. The slides PDF is one of the room's files, so on the class page it is ticked from the start and opens for students in the browser's own viewer. Handwritten annotations and speaker notes do not go onto the page. Before the next lecture, make sure the right room is open on every device. If the tablet has lost its rights, sign in with a new console link; do not share your admin sign-in link to restore access."},{"title":"Class pages and courses","section":"Run a class","url":"publishing.html","text":"Put a class's materials on a permanent page and give the group one link for the whole course."},{"title":"Three addresses, three purposes","section":"Class pages and courses","url":"publishing.html#links","text":"Link What it opens /s/… The live room: notebooks, files, edit history, marks and questions to the oracle. What people can do depends on the room rules and on whether the class has ended. /p/… The class page: a folder of the class's materials — notebooks with their run results, PDFs, data, and everything as one archive. No kernel and no shared editing. /c/… A course: a timetable by day, with links to the class pages. The page and the room are two different things A page is built from a room, and after that it does not change with what happens in the room until someone updates it. The room stays the place where the class happened: it keeps the edit history, the marks and the conversation with the oracle. None of that goes onto the page, but the page can lead back to the room . Publishing does not close the room. To leave the group read-only access, click “End class” in the room (“End the class” in the panel)."},{"title":"Publish a class","section":"Class pages and courses","url":"publishing.html#publish","text":"Materials are chosen on the Class page screen. There are three ways to it: after the class ends, if the room is in a course and has no page yet: the room shows a “Class finished” window; Choose materials → opens the screen in a new tab, and “Not now” closes the window until the next time the class is ended; from the strip of a finished class in the room: the link “Publish materials →”, which becomes “Class page →” after the first publish. Only the teacher sees it, and every room has it, in a course or not; from the panel: Class page… in the class menu under “Classes”, or in a row's menu on the course page. Check the ticks under “Notebooks” and “Files”. Colloq sets them itself and writes the reason next to every unticked row (see below ). Rename materials if needed. A notebook's name becomes its tab (“Lecture”, “Seminar”); Markdown text ( .md ) opens as a tab too, with its own table of contents; a PDF's name becomes the label of its link; data, code and folders keep their room path, because the code refers to it. The ↑↓ arrows change the order of the notebooks. If a “Check” section appears, look at every warning and tick “Checked — publish as it is”. Click Publish . The screen shows the address of the finished page with “Copy link” and “Open”. Open it the way a student will. A page holds up to 24 materials and up to 200 MB in total, and each file can be at most the server's upload limit (50 MB by default). A folder is one material however many files it holds (up to 1,000). The screen shows how much is used: “Page: 10.8 MB of 200 MB”. Publishing again — “Update the page” on this screen or in a course row's menu — rebuilds the page at the same link from what is in the room now. Updating never brings a withdrawn page back: its course row has no such item, and on the Class page screen the button says “Publish again”. If a ticked file is gone, has grown over the limit, or the page is full, the screen names it under “Left off the page”. A folder is only ever left off whole: half a dataset is worse than none."},{"title":"What is ticked from the start","section":"Class pages and courses","url":"publishing.html#pick","text":"Every notebook is ticked except empty ones and those that look like someone else's or private. PDFs are ticked unless one of the reasons below applies: they are usually the slides. Data, code and text files are ticked only when a notebook refers to them (the row says “used in seminar.ipynb”) and they are small: data up to 20 MB, code and text up to 200 KB. A README at the top of the room is ticked even without a reference. The ticks are suggestions, not decisions: any unticked row can be ticked, except an empty notebook and a file over the upload limit. The rules lean towards leaving things out: adding a file is one click, while a link already handed out cannot be taken back. Reason on the row When it is given “a student's notebook · Akim Zuev” A student's own notebook from this room. “looks like a student's work” The file name contains the name of someone in the class, for example ZuevAkim_02.ipynb . “the name starts with «_»” The file, or a folder on its path, starts with an underscore: a handy way to hide drafts. “looks like answers” The name has answers, solutions or a similar word, in English or Russian. “empty” The notebook has neither text nor output. It cannot be ticked. “not mentioned in the notebooks” No notebook names the file or its folder. “looks like a run's output” The file is in outputs/ , results/ , logs/ , checkpoints/ or a similar folder, or it is model weights, a log, an HTML file over 1 MB or a submission… file. “the picture is already in a notebook's text” The picture is embedded in a text cell and goes onto the page with the notebook. “over …” The file is over the threshold for its kind. Over the upload limit, it cannot be ticked. “looks like a key or a password” A file in a folder is named like a key: id_rsa , *.pem , credentials.json , token.txt . “cannot be checked for keys — open it and decide” The check cannot read a file in a folder: HTML, a file without an extension, code or text over 2 MB. “over 1000 files” The folder holds more than 1,000 files. It cannot be ticked. “nothing inside can go out” Every file in the folder stays in the room by the rules in this table. Notebooks and files that appeared after the previous publish are marked “new” and left unticked: neither chosen nor knowingly left out."},{"title":"Folders: data, modules and pictures","section":"Class pages and courses","url":"publishing.html#folders","text":"A top-level folder of the room — data/ , scripts/ , assets/ — is chosen whole, as one row: “data/ · 15 files · 25 MB”. On the class page it is one row too, “Data · 15 files · 25 MB”, and downloads as one ZIP with the folder inside. This is how the modules the notebooks import ( from scripts import case_cian as cian ) and the pictures in their text end up beside them, at the same paths as in the room. A folder is ticked from the start when something being published uses it: a ticked notebook reads a file in it or imports it as a package (“read in lecture.ipynb and scripts/eda_tools.py”, “imported in lecture.ipynb”), code in another ticked folder reads it, or the notebooks' text shows pictures from it. An unticked notebook, such as solutions.ipynb , ticks no folder. Inside a folder the rules from the table above apply, except “not mentioned in the notebooks”: drafts starting with «_», answers, a participant's name anywhere in the path ( hw/zuev_akim/model.py ), run outputs and files over the upload limit stay in the room. So do files the check cannot read: those named like a key and files of an unknown kind. Data and pictures go at any size within the limit. __init__.py always goes, and a package's «_» module goes when the published code imports it ( from ._utils import plot ): without it the package does not import after unpacking. The what is inside ▾ link under the row lists every file with a tick, and a reason next to those that stay in the room. The ticks can be changed: tick _utils.py or submission_example.csv , untick holdout.csv . Only a file over the upload limit cannot be ticked. If by the rules nothing in a folder can go out, tick the file you need in its contents, and the folder is ticked with it. A folder with more than 1,000 files, such as lib/ after pip install --target , is not published: “over 1000 files”. If the server is rolled back to 0.13 or an earlier release, at startup it deletes from the page store the files only folders refer to. Back on the current release, Colloq puts them back at startup from the room when the room still has the same bytes; a folder it cannot restore that way is taken off its page, with a line in the log. The folder stays chosen: “Update the page” or the next end of class brings it back from the room."},{"title":"The check before publishing","section":"Class pages and courses","url":"publishing.html#check","text":"A room's address is the right to write into it, so it is always removed from cells and output, without asking: the path /workspace//data.csv in a traceback becomes data.csv . The screen only reports it: “The room's address was removed from 2 cells; it will not be on the page.” Everything else Colloq does not fix but shows under “Check”: “Looks like an API key” — in a cell, its output or a ticked file, with the notebook and the cell number or the file name, and the key, partly masked; “Looks like a student's name” — the full name of someone in the class in a cell, its output, a plotly chart or a ticked file; a first name on its own does not count; “The room's address is in …” — a text file is published as it is, and the address in it is not changed. Code, text and data files up to 2 MB are checked the same way as cells: a utils.py with a key, a kaggle.json or a class list in a CSV goes onto the page byte for byte, so it is shown under “Check” rather than fixed. A string that looks like a key may be a placeholder the lesson is about, and a full name may be a scientist's, so the teacher decides. “Open in the room” leads to the cell: fix it there, or tick “Checked — publish as it is”. Until it is ticked, the publish button stays disabled. The confirmation is remembered for that finding; a new key or name is a new finding."},{"title":"Updating every time the class ends","section":"Class pages and courses","url":"publishing.html#refresh","text":"The choice is made once; after that, ending the class is enough. Update the page every time the class is ended is on by default: the ticks are remembered, and each time this room's class is ended again, Colloq rebuilds the page by itself. Queued cells keep running after the class ends, so Colloq first waits for them to finish, for at most two minutes. The teacher in the room sees “The page will update once the cells finish running”, then “Class page updated · 4 materials”. New notebooks, files and folders are not added by themselves: next time they wait on the screen marked “new”. A ticked notebook that was deleted from the room is skipped. A ticked folder is read again whole: a new file in data/ goes onto the page by the folder rules and passes the same check. Files whose tick in “what is inside” was changed by hand keep that decision. If something new turns up in the check — a key, a name, or the room's address in a file — the page stays as it was, and the room shows “The page was not updated: …” with a “Review” button. “Resume class” never touches the page. If the class is resumed, the page withdrawn or the option switched off while the cells finish, there is no update, and its notice goes away. When the class is ended from the panel, the panel says the same in short: “The class is over · the class page is being updated.” To stop the page from changing by itself, untick the option and publish again."},{"title":"What becomes public","section":"Class pages and courses","url":"publishing.html#included","text":"The Class page screen shows the same list above the publish button. Published Not published Ticked notebooks — code, text and run results: plots, tables, messages, tracebacks and pictures in text cells. Each notebook downloads as .ipynb together with its results and pictures. Who wrote and ran what. Ticked files and folders — as they are in the room's folder. A PDF opens in the browser's own viewer, a folder downloads as one ZIP, everything else as it is. Questions to the oracle — they stay in the room. The “Download all” archive: one folder with the materials and every file of the ticked folders at the same paths as in the room, and a README with the title, the day, the links and the list of folders. After unpacking, pd.read_csv('data/train.csv') and from scripts import case_cian run unchanged. The terminal, unticked notebooks and files, the participant list, the room's address in cells and output. A name, a key or personal data already written in code or output becomes part of the page, and the check does not catch everything. Look through the ticked notebooks yourself. The page asks search engines not to index it. This is not access control: anyone who has the link can open the page and pass it on. When many people download one class's archive at once, the server sends it to three at a time (and six across the whole server), and the rest wait their turn: the download simply starts a little later. Only after more than a minute of waiting does “Too many downloads — try again in a minute” come back. A download that stands still for a minute is cut off and frees its place."},{"title":"The class day","section":"Class pages and courses","url":"publishing.html#day","text":"Every class in a course has a day — a calendar date in the server's time zone. The course page uses it to sort classes into months, draw the “today” line and decide which class was the previous one and which is next. It is the day of the course row, not the publication date: materials put up a week before the class still sit on its day. The day appears by itself: from the plan : a row has a “Class day” field, and a new row defaults to the previous row's day plus a week. A room created from a row, or assigned to it, takes over the day along with the topic; from attendance : on the upgrade to this release, plan rows whose week text names a single day (“Sun, Sep 6”, “06.09”) got that day, and classes already held got the day the most students joined the room, otherwise the day the class was ended; when the class ends : a row without a day gets the day the class was ended. A week written as text (“31 Aug – 6 Sep”) is shown until a day is set. To correct a day, use the course: the row's menu → “Edit the row…”. For a page outside a course, the day is the “Class day” field on the Class page screen; by default it is the day the class was ended, otherwise the day of the first publish."},{"title":"Build a course","section":"Class pages and courses","url":"publishing.html#course","text":"In the panel, open Courses and create a course: its title, description and address. The term plan is typed in with “+ Planned topic”: the topic is required, and next to it are “Class day”, “What it is about — a sentence or two” and the “A break — no number” box for holidays. “Add to plan” or Enter saves the row, and the form stays open for the next one. The easiest way to create a class's room is from the course: “Create a room” on a plan row, or the “Course class” field on the “New class” screen, where the open rows of every course are listed nearest first. The room takes its row, with the topic, day and summary already there. An existing room goes into a row with “Assign class”, and “+ Add a class” puts it at the end of the course. Students see the row's topic, or for a room added without a plan row, its name; “Title for students” under “Edit the row…” sets a title of its own. The “Page” column of the course table shows where each class stands: “planned”, “page · 4 materials”, “before class · 2 materials” when the materials went up early, “withdrawn”, and for a past class with no page the warning “The class is over and has no page — Publish…”. Under the page address, Edit the materials opens the class page, where materials are added, removed and published again. The row's menu has “Class page…”, “Update the page”, “Page address…”, “Open as a student” and “Take the page down”. Give the group the /c/… address once. On the course page a student sees: the “now” blocks: “Today · Sun, 4 Oct” or “Next · Sun, 4 Oct · in 2 days”, and “Previous class · Sun, 20 Sep”, with links to the materials and the archive. Before the course starts the top block says “First class”, after it ends “Last class”, and the header adds “course finished”; the “Classes” list in course order, with month labels (“September”, “January 2027”) and a “Today” line between past and future classes. A class with a page has a link and a summary of what is on it: “lecture · seminar · slides · data”; a future class shows its day; on the day itself it says “today · materials after the class”; materials put up early are labelled “materials before the class”; breaks without a number: “Holidays — no classes”; classes more than three months ahead folded under “show the whole plan”, when there are more than eight of them. “Today” is the server's day, not the student's browser's. If a past class has no page, the course says “No materials yet.” and offers the latest published materials. The course page does not reveal room addresses, except for the “Enter the room” button on the day of a class (see below )."},{"title":"Entering the room from the page","section":"Class pages and courses","url":"publishing.html#room","text":"The room keeps what the page does not have: the edit history, the marks and the questions to the oracle with their answers. To make it easy to get back to them, the class page has a Class room block under the materials with a “Go to the room” button; on a phone it is a “Class room” row under the archive button. On the day of a class, while it is on, the “Today” block of the course page becomes “Today · Sun, 4 Oct · class in progress” with an Enter the room button. Who sees the way in is chosen by the teacher on the Class page screen, under “Entering the room from the page”. Once the page exists, the choice is saved at once and does not rebuild the page; before the first publish it goes out with it: Option Who sees the way in “Everyone” (default) Anyone who has the link to the page or the course. A note under the button says that the room shows the participants’ names and their questions to the oracle. “Participants” Those who joined this room from this browser and were not removed from the class, and the teachers of this room. Everyone else reads: “The room is open only to those who were in the class. If you were, open this page in the browser you joined from.” “No one” The teachers of this room only: the page hides the way in. The button leads into the room itself, and the room's rules apply there: while the class is on, whoever comes in works as an ordinary participant; after it ends, they can only read. A course row without a page lets people in under the “Everyone” rule: during a class, anyone with the course link can enter the room. A withdrawn page or a deleted room has no way in for anyone, teachers included. Pages carried over from earlier releases also lead into the room under the “Everyone” rule; another choice is kept from their next publish from this screen. Pages published in 0.13 keep the choice saved back then: “Participants”, unless the teacher changed it. To open their room to everyone, switch the way in to “Everyone” by hand. Why “Everyone” by default Students have no accounts, so a “participant” is recognised only by the browser they joined from. The same student on a phone, in another browser or after clearing site data is a stranger to the room, although the page is made for exactly them. The cost is that a room is not a page: it shows who wrote what, participants’ names and their questions to the oracle with the answers, and pages get forwarded. If the group does not want every reader of the page to see that, choose “Participants” or “No one”. Under “Participants”, another browser or a private window shows no button, and a newcomer gets the room link from the teacher."},{"title":"Short and previous addresses","section":"Class pages and courses","url":"publishing.html#addresses","text":"A class page or course can get a name made of lowercase Latin letters, digits and hyphens, 3 to 64 characters long, starting and ending with a letter or digit. For example, /c/python-autumn . The first publish from a course with such a name gets an address like /p/ml-strong-04 (the course name and the class number) if it is free; renumbering the rows never changes it afterwards. A page's address can be changed at any time — on the Class page screen, or with “Page address…” in a course row's menu. After a rename, the previous address keeps leading to the current material and stays reserved. Releasing the previous name stops the old link from working and lets the name be used again. Release it only when you are ready to change an address you have already handed out."},{"title":"Take a page down or delete a room","section":"Class pages and courses","url":"publishing.html#remove","text":"“Take the page down” in a course row's menu (or in the class menu under “Classes”) closes the page: the link tells students the teacher took it down. “Put the page back” in the same place, or on the Class page screen, opens it again. Publishing and archiving do not change the permissions in the live room. By default, deleting a room keeps its page; read the options in the delete confirmation carefully. The materials are stored apart from the room, so the page survives the deletion and its course row keeps leading to it. Such pages are collected on the “Courses” page under “Pages without a room”: you can withdraw them (“Withdraw page”) or bring them back (“Restore page”), and the owner can delete a withdrawn page permanently. On the Class page screen such a page keeps only its current materials: you can “Take off the page” what is not needed, its images included, but there is nothing left to add from. Page files live in data/page-files/ and are part of the server backup ; a file used on several pages is stored once. But a page does not replace a backup: it does not contain all the working files, configuration, secrets and room data."},{"title":"Older pages and links","section":"Class pages and courses","url":"publishing.html#old","text":"Pages published before class pages existed were turned, on the first start of the new release, into pages with one material, “Notebook”: the last published state of the notebook with its results, contents and archive. Pages no longer have steps or marked moments; the marks in the “History” panel stay in the room as restore points. Links already handed out keep working: /p/
    / opens the class page, and the old notebook download link leads to the download of the page's first notebook, now with its run results. GitHub import now also takes PDF and Markdown files: slides and READMEs kept in a repository reach the room, and from there the class page. The static export of a course into a repository ( make site ) is gone: course and class pages live only on your server, and an offline copy of a class is its “Download all” archive."},{"title":"Colloq in your Kubernetes","section":"Your own server","url":"kubernetes.html","text":"A Helm chart in one namespace of a corporate cluster: permissions, security, network, storage, a registry mirror, installing with Helm or ArgoCD, and operations."},{"title":"In short for the platform team","section":"Colloq in your Kubernetes","url":"kubernetes.html#summary","text":"What it is Colloq is a collaborative Jupyter classroom: a web app, a runtime broker and a separate Pod with a Python kernel for each room. It goes into your cluster as a Helm chart, oci://ghcr.io/colloq-edu/charts/colloq , in one namespace. Open source, MIT license. What runs The app Deployment , one replica: the database is SQLite. The broker Deployment , one replica: the only thing that talks to the Kubernetes API. A Pod and a Service for each running room. A second Pod and Service per room for students' personal notebooks. Short-lived competition Pods: submissions, the metric, package preparation. Room and competition Pods are created by the broker, not by the chart. What it needs One namespace with namespace-admin rights; the broker's Role from Permissions ; a StorageClass — ReadWriteOnce required, ReadWriteMany preferred; an ingress controller and a certificate; a registry mirror with the release's images. What it never needs cluster-admin or cluster-scoped objects: Namespace, PersistentVolume, ClusterRole, CRD, DaemonSet, RuntimeClass. Privileged containers, hostPath, hostNetwork or hostPID. Root: every container runs as uid 1000. Internet access from the cluster. Network Inbound: only the ingress controller to the app, port 3000. Inside the namespace: app → broker, app and broker → rooms. Outbound: broker → Kubernetes API, Pods → cluster DNS; optionally app → the AI model and an outbound proxy. Students' code gets DNS only; competition submissions get nothing. Data Two PVCs: colloq-data — the SQLite database, keys, class page files, competition data and submissions, database snapshots; colloq-workspace — room files. Backups Volume snapshots with your own tools (Velero, CSI), plus consistent database snapshots the app writes into the colloq-data volume itself once a day. Students' and teachers' browsers ↓ HTTPS The cluster's ingress controller, TLS ↓ :3000 App · Deployment colloq-app, 1 replica ↓ :8787, Bearer Broker · Deployment colloq-runtime, 1 replica ↓ Kubernetes API: create and delete Pods Room A · Pod and Service Room B · Pod and Service The room's personal notebooks Competition submission The app — a Node.js server with the built web interface and an SQLite database — keeps everything on two volumes and never calls the Kubernetes API. When someone joins a room or runs a cell, the app asks the broker for the room's Pod. The broker creates the Pod and a ClusterIP Service from a template fixed in its code, waits for Jupyter and hands the app its address and token; from then on the app talks to the room's Jupyter directly, on port 8888. A room's Pod lives while anyone is in the room and for 2 hours after everyone has left. Room Pods survive restarts and upgrades of the app and the broker: the new broker finds them by their labels, and Python variables stay. The security boundary Students' code runs in room Pods: uid 1000, no API token, no Linux capabilities, no network but DNS. Only the broker has rights in the cluster, and only in this namespace: to create and delete Pods and Services. Even a compromised broker can only create a Pod that restricted Pod Security and your policies admit; but such a Pod can mount Colloq's volumes, so compromising the broker means access to all of this installation's data, though nothing beyond its namespace. Pods share the node's Linux kernel: this is not VM-level isolation."},{"title":"What we assume about your cluster","section":"Colloq in your Kubernetes","url":"kubernetes.html#assumptions","text":"This is a checklist for your team: go through it before installing. The answer to each item is either \"yes, that's us\" or a chart value to set. What We assume If yours differs Kubernetes version 1.30 or newer, upstream-compatible: vanilla, Deckhouse and the like. Changing a running room's memory and cores without a restart needs 1.33 or newer and the pods/resize permission. Older than 1.33, or no such permission: runtime.inPlaceResize: false . The rule is dropped from the Role, resizing a live room answers \"unsupported\" rather than failing, and the new number goes to the room's next Pod. Namespace We get one namespace with namespace-admin rights, and it holds one Colloq installation: object names are fixed ( colloq-app , colloq-runtime , …). The chart creates no cluster-scoped objects. Your team creates the namespace, its labels and its quota. The chart creates the Role and RoleBinding itself, and Kubernetes won't let anyone grant a verb they don't hold: whoever installs the chart needs every verb in Permissions . Pod Security and policies Pod Security Admission restricted is enforced, and Kyverno or Gatekeeper require requests and limits on every container, probes, runAsNonRoot and readOnlyRootFilesystem, forbid privileged, hostPath and hostNetwork, admit images only from the internal registry and only by digest, and require the standard app.kubernetes.io/* labels. Every Colloq Pod passes restricted ; details and exceptions are under Security . commonLabels adds your labels to every chart object and every broker Pod; rooms.podLabels and rooms.podAnnotations add them to broker Pods only. Network policies NetworkPolicy is enforced (Calico, Cilium), and the namespace may be default-deny. The chart declares every flow it needs, ingress and egress; they are listed under Network . Say where the ingress controller lives ( networkPolicy.ingressController.from ), where cluster DNS lives ( networkPolicy.dns.to ), and the API servers' addresses ( networkPolicy.kubeApiServer ). Ingress and TLS Ingress through ingress-nginx, with a configurable class and annotations. TLS: the organisation's certificate from an existing Secret or from cert-manager. Users come from the corporate network, over VPN or, when students are external, from the internet. Another controller: ingress.className and ingress.annotations with the same meaning — request bodies of 256 MB or more, timeouts of an hour or more, no response buffering. Or ingress.enabled: false : publish the Service colloq-app:3000 with your own object and set config.publicUrl . Sign-in Students join with the class link, teachers sign in with a personal link. A proxy with SSO that forwards a signed JWT — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — may stand in front of Colloq. config.sso.* : teachers sign in with their corporate account, students who have one join under their own name; see Sign-in through your SSO . Without such a proxy, nothing to set. Storage A StorageClass with ReadWriteOnce always exists; one with ReadWriteMany (CephFS) may. By default the chart assumes ReadWriteOnce: everything that mounts Colloq's volumes runs on one node. With ReadWriteMany, set persistence.accessMode: ReadWriteMany and room Pods spread over the cluster; see Storage . No internet The cluster has no internet access. Images come through a registry mirror (Harbor, Nexus), PyPI packages through a Nexus mirror, and the Oracle through an internal model with an OpenAI-compatible API, or it stays off. There may be an outbound HTTP proxy and an internal CA. global.imageRegistry — the mirror for every image, kernel images included; dependencies.indexUrl and dependencies.mirrorEgress — the PyPI mirror; config.ai.baseUrl — the model; outbound.httpsProxy , outbound.noProxy , outbound.extraCa . See Images and the registry mirror . Secrets and GitOps Secrets come from Vault through External Secrets, and deployment goes through ArgoCD. Every secret can come from an existing Secret: config.existingSecret , secrets.runtimeToken.existingSecret , secrets.roomSecret.existingSecret , metrics.existingSecret . With those set, the chart renders no random value, so ArgoCD sees no perpetual drift. Monitoring and logs Prometheus Operator; logs are collected from stdout and stderr. metrics.enabled turns on /metrics with a token, metrics.serviceMonitor.enabled adds a ServiceMonitor; without the Prometheus Operator, scrape /metrics your own way. config.logFormat: json writes one JSON object per line. Backups Volume snapshots: Velero, CSI. The app itself writes consistent database snapshots into the colloq-data volume ( config.dbSnapshotHours , config.dbSnapshotKeep ), so a volume snapshot always carries a whole copy of the database. GPU GPU nodes are optional. GPU rooms take a RuntimeClass, a nodeSelector and tolerations: gpu.runtimeClassName , gpu.nodeSelector , gpu.tolerations ; see Operations ."},{"title":"Permissions","section":"Colloq in your Kubernetes","url":"kubernetes.html#permissions","text":"Only the broker has rights in the cluster: the Role and RoleBinding colloq-runtime in this namespace. No ClusterRole , no access to nodes, Secrets, Deployments or other namespaces. Resource Verbs Why pods get, list, create, delete Room, personal-notebook and competition Pods. list is for the census: a restarted broker finds the running Pods by their labels and adopts them instead of creating them again. services get, list, create, delete A ClusterIP Service for each room and its personal notebooks — the stable name the app reaches Jupyter by — and for the package-preparation proxy. A room deleted for good leaves a tombstone: a headless Service with no selector and no ClusterIP, so that its ID can't be opened again. pods/resize patch Change a running room's memory and cores without a restart (Kubernetes 1.33+): the patch touches only memory and cpu of the kernel container. Optional: with runtime.inPlaceResize: false the rule is gone. persistentvolumeclaims get Before a competition, check that the data volume is in place. networkpolicies (networking.k8s.io) get Before a competition, check that the submissions' isolation policies exist. Without them submissions are refused, and the teacher sees why. There is no patch or update on Pods and Services: the broker doesn't edit live objects — it changes only a room's resources through pods/resize , and deletes and recreates everything else, with a UID precondition. RBAC can't constrain the fields of a Pod, so the Pod templates are fixed in the broker's code, and its HTTP API takes only a room ID, an environment from the catalog, an image revision, memory and cores: no arbitrary template, no hostPath, no image outside the catalog. Restricted Pod Security and your policies hold the rest. ServiceAccount API token Who colloq-app no The app. It never calls the Kubernetes API. colloq-runtime yes: the broker's Pod mounts a projected token that the kubelet rotates The broker; bound to the Role above. colloq-kernel no Room and personal-notebook Pods: students' code holds no API credentials. the namespace default no Competition Pods: no token is mounted. All three ServiceAccounts set automountServiceAccountToken: false ; only the broker's Pod turns the token on."},{"title":"Security","section":"Colloq in your Kubernetes","url":"kubernetes.html#security","text":"Pod Security. Every Colloq Pod — the app, the broker, rooms, personal notebooks, competitions — passes the restricted profile: runAsNonRoot , uid and gid 1000, seccomp RuntimeDefault , allowPrivilegeEscalation: false , capabilities.drop: [ALL] , readOnlyRootFilesystem: true . Only volumes are writable: emptyDir for /tmp , the home directory and /dev/shm (in memory: 64 MiB, 1 GiB for GPU rooms), and the PVCs. Your team puts the pod-security.kubernetes.io/enforce: restricted label on the namespace: the chart doesn't own it. Policy engines. Requests and limits are set on every container, including the init container and the sidecar of competition Pods. A room kernel's requests equal its limits (QoS Guaranteed): memory and cores are reserved in full, plus 2 GiB of ephemeral storage. Probes. The app: HTTP startupProbe and livenessProbe on /api/livez , readinessProbe on /api/readyz . The broker: TCP 8787. Room and personal-notebook kernels: TCP readiness and liveness probes on 8888; the liveness probe acts only after five minutes of refused connections, since a Pod with restartPolicy: Never ends with the class's variables when it does. Submission containers run to completion and have no probes; their exporter and the package proxy have readiness probes. If a probe rule doesn't fit run-to-completion Pods, exempt them by the label app.kubernetes.io/managed-by: colloq-runtime . Labels. Chart objects carry app.kubernetes.io/name: colloq , instance , component , version , part-of , managed-by: Helm and helm.sh/chart . Pods the broker creates carry app.kubernetes.io/name , instance and part-of , app.kubernetes.io/managed-by: colloq-runtime and colloq.dev/role : kernel (the personal-notebook Pod also has colloq.dev/kernel: own ), competition-job , competition-resolver , competition-proxy . The app.kubernetes.io/* and colloq.* keys belong to the chart and the broker and can't be overridden. Digests. The chart always renders the app and broker images as registry/path@sha256:… — never a tag, never latest . Kernel images in the catalog are accepted only in the same form: neither the chart nor the broker lets a tag-only reference through. Registry. global.imageRegistry rewrites the registry of every image, including the catalog's kernels, the competition exporter and the package proxy. Mutating webhooks. After creating a room's Pod the broker checks it against what it asked for. If a webhook changed its containers, volumes or security settings — injected a sidecar, rewrote imagePullPolicy — the broker refuses the Pod and the room doesn't start; the broker's log names the fields that changed. Exempt Pods labelled app.kubernetes.io/managed-by: colloq-runtime from such webhooks; for Istio, rooms.podLabels: {sidecar.istio.io/inject: \"false\"} is enough. Placement the cluster fills in by itself — PodNodeSelector, PodTolerationRestriction, priority, RuntimeClass, Kueue — doesn't disturb the check. What code runs as whom. Pod Image and code API token Volumes Network App colloq-app : the Colloq server on Node.js no colloq-data and colloq-workspace in from the ingress controller; out to the broker, rooms, DNS and, optionally, the AI model and a proxy Broker colloq-runtime : the broker on Node.js yes none in from the app; out to the Kubernetes API, rooms, submissions and DNS Room the room environment's colloq-kernel : Jupyter and students' code no its own folder of colloq-workspace in from the app and the broker on 8888; out: DNS only Personal notebooks colloq-kernel : students' code in personal notebooks, no GPU no its room's folder, at the same path as a room Submission colloq-kernel : the entrant's notebook or the metric; colloq-runtime : the result exporter no folders of colloq-data , inputs read-only none; in: only the broker to the exporter on 8765 Package preparation colloq-kernel : pip downloads entrants' own packages no folders of colloq-data only to its own proxy on 3128, and DNS Package proxy colloq-runtime : a CONNECT proxy no none in from preparation; out: DNS and the package index — public PyPI on 443 or your mirror Isolation between rooms. Each room has a Pod of its own: its own processes, its own emptyDir , its own Jupyter token — the broker derives it from the room secret with an HMAC and gives it to the app only. The Pod mounts only its own folder of the colloq-workspace volume ( subPath ); other rooms' files aren't in it. The network policy lets only the app and the broker reach a room's port 8888 and lets nothing out of a room but DNS: neighbouring rooms, the app and the API are out of reach of students' code. Inside a room, participants share the kernel, the files and the Linux user — by design: it is a shared workspace. Standard NetworkPolicy doesn't guarantee that a Pod can't reach services on its own node: test it from a room (see the checks after installing ) and close it with your CNI's host policies if needed. Secrets. Secret What it holds Who reads it config.existingSecret or colloq-app-config Keys are app settings: OPENAI_API_KEY , SESSION_SECRET , a proxy with a password in HTTPS_PROXY , or any other setting the app colloq-runtime-auth , key runtime-token The Bearer token the app uses with the broker the app and the broker colloq-room-secret , key room-secret The key the broker derives rooms' Jupyter tokens and submission export tokens from the broker only colloq-metrics , key metrics-token The Bearer token for /metrics , when metrics are on the app and Prometheus global.imagePullSecrets The mirror's .dockerconfigjson ; broker Pods get the first one too the kubelet Each can come from an existing Secret — for example one External Secrets creates from Vault. When the broker token, the room secret or the metrics token are not given, the chart generates them: under the helm CLI once, reading them back from the cluster with lookup afterwards. ArgoCD and other tools that render the chart with helm template can't read them back and would generate new values on every sync, restarting rooms — set existingSecret there. The broker token and the room secret are 43 to 128 characters of [A-Za-z0-9_-] from at least 32 random bytes: openssl rand -hex 32 . Keep the secrets stable: a new SESSION_SECRET invalidates every link and sign-in handed out, and a new room secret replaces room Pods at their next start, losing Python variables. Without SESSION_SECRET , the app generates one once and keeps it in the colloq-data volume. When you move from a generated secret to existingSecret , copy its value into the new Secret first: the chart deletes its own. Secrets live in the colloq-data volume too: the setup token, the teachers' sign-in keys in the database, the model key if it was entered in the panel. Keep snapshots of that volume as private as Secrets."},{"title":"Network","section":"Colloq in your Kubernetes","url":"kubernetes.html#network","text":"Every flow Colloq needs. The chart declares them as network policies, ingress and egress, so the installation works in a default-deny namespace too. From To Port Why Ingress controller App TCP 3000 All user traffic: pages, API, WebSocket, server-sent events. networkPolicy.ingressController.from says who is admitted (by default the ingress-nginx Pods in the ingress-nginx namespace). A controller on the host network arrives from node addresses: give an ipBlock of your nodes then. App Broker TCP 8787 Start, resize and stop a room's Pod, start a submission. Requests carry a Bearer token. App Room and personal-notebook Pods TCP 8888 Jupyter: running cells, the terminal. With the room's token. Broker Room and personal-notebook Pods TCP 8888 Check that Jupyter is up and accepts the token. Broker Submission and package-preparation Pods TCP 8765 Collect the result from the exporter. Broker Kubernetes API server 443 and 6443 Create and delete Pods and Services. The network policy sees the address after DNAT, that is the API servers' own addresses, so by default any address on these ports is allowed. Narrow networkPolicy.kubeApiServer.cidrs to the output of kubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes . On Cilium the API server has an identity of its own that an ipBlock doesn't match: turn on networkPolicy.kubeApiServer.ciliumEntity , and the chart adds a CiliumNetworkPolicy in this namespace. Every Colloq Pod except submissions Cluster DNS UDP and TCP 53 Service names. networkPolicy.dns.to says where DNS is (by default k8s-app: kube-dns in kube-system ). Rooms can lose DNS too: rooms.network: none . Package preparation Its own package proxy TCP 3128 pip goes only through a proxy that admits one package index. Package proxy PyPI mirror or public PyPI 443 or the mirror's port Index and files of competition entrants' own packages. A policy knows addresses, not names: the mirror's addresses and ports go into dependencies.mirrorEgress . In a closed network turn off the general rule for public addresses on 443: dependencies.publicEgress: false . App AI model, outbound proxy, GitHub the model's or proxy's port Optional: the Oracle and notebook import from GitHub. By default the app may go nowhere outside: addresses and ports go into networkPolicy.app.egress . Prometheus App TCP 3000, /metrics Optional, with metrics.enabled : networkPolicy.monitoring.from says who is admitted. Nodes (kubelet) Registry mirror TCP 443 Images. Not described by network policies. If the cluster runs NodeLocal DNSCache, DNS queries go to a node-local address rather than to the kube-dns Pods: add that address to networkPolicy.dns.to , for example ipBlock: {cidr: 169.254.20.10/32} . Students' code doesn't get out: room and personal-notebook Pods have no egress but DNS, and rooms.network: none takes that away too. So pip install in a cell doesn't work: libraries come in kernel images; see Images and the registry mirror . Competition submissions get no network at all, not even DNS, and that is checked before any code runs: an init container tries to get out and lets the submission run only after three refusals in a row; if the network answers, the submission fails. The client address: TRUSTED_PROXIES=private . Per-address caps — 60 newcomers to a room per 10 minutes, 20 Oracle questions a minute, competition sign-ins and joins — must see the student's address, not the ingress controller's. The app believes X-Forwarded-For only from connections whose address is in TRUSTED_PROXIES ; the chart sets private ( config.inbound.trustedProxies ) — every private range, wherever the controller's Pod comes from. That is safe because only the ingress controller is admitted to the app's port 3000: the network policy lets no one else in, students' code included, so nobody inside the cluster can forge the header. The client is the rightmost address in the header that is not itself trusted; when every hop is trusted — and on a corporate network users' addresses are private too — the leftmost stands, which is the one the controller wrote. So the controller must replace the header rather than append its own address to one the client sent: ingress-nginx does that by default, while use-forwarded-headers and compute-full-forwarded-for are off. If a load balancer in front of the controller rewrites the source address, every student looks like one address: preserve the client address ( externalTrafficPolicy: Local on the controller's Service, or PROXY protocol). If many people come from one address — through a NAT or a VPN concentrator — list it in config.inbound.sharedAddresses : per-address caps don't apply to it, while per-room and per-person caps still do. ingress-nginx. The chart puts these annotations on its Ingress ( ingress.annotations ; a key set to null drops an annotation): nginx.ingress.kubernetes.io/proxy-body-size: 256m # competition data: up to 200 MB in one request nginx.ingress.kubernetes.io/proxy-read-timeout: \"3600\" # WebSocket and server-sent events live for hours nginx.ingress.kubernetes.io/proxy-send-timeout: \"3600\" nginx.ingress.kubernetes.io/proxy-buffering: \"off\" # events must arrive one by one Without the first line ingress-nginx accepts no more than 1 MB, and uploading competition data or a file to a room gets a 413. The controller passes WebSocket ( /collab/ , /control/ , /file/ ) by itself; the server pings every 25 s, and server-sent event streams send a heartbeat every 20 s and the header X-Accel-Buffering: no . The app serves the root of its own hostname and won't work from a sub-path: the chart's Ingress is by the host ingress.host , path / . A file uploaded to a room is up to 50 MB by default ( config.maxUploadMb ); raise that and raise proxy-body-size too. TLS and the address. The organisation's certificate is an existing Secret of type kubernetes.io/tls : ingress.tls.secretName , colloq-tls by default. Or cert-manager: ingress.tls.certManager.clusterIssuer adds the annotation with your issuer. PUBLIC_URL is the external address every link Colloq hands out is built from, the owner link included; by default the chart takes https:// , and config.publicUrl sets another. When the controller says X-Forwarded-Proto: https , cookies get the Secure flag and the app sends Strict-Transport-Security . ingress-nginx adds that header itself by default; to avoid sending it twice, turn the app's off: config.inbound.hsts: false . The ingress controller's access logs contain room addresses, staff sign-in keys ( /admin/k/… , /admin/t/… ) and room tokens in WebSocket addresses ( ?token=… ). Keep them as private as the database."},{"title":"Sign-in through your SSO","section":"Colloq in your Kubernetes","url":"kubernetes.html#sso","text":"If a proxy in front of Colloq checks the person itself and forwards a signed JWT with every request — Teleport application access, oauth2-proxy or Pomerium in front of your IdP — Colloq accepts that sign-in. The signature is checked against the proxy's public keys (JWKS), an expiry and the audience are required, and the issuer is checked when set. The audience is required because one proxy signs the tokens of all its applications: without it, a token issued for another application would sign in here too; * accepts any. Only asymmetric algorithms are accepted (RS, PS, ES, EdDSA); none and HMAC are always refused. A request without a token, or with one that does not verify, is served as before: links keep working, and the proxy cannot lock a class out. Teachers. A token whose email is on the staff list signs in as that teacher: Colloq issues the ordinary cookie, and the panel, the room and the sockets see the signed-in teacher. The owner still adds colleagues by email, but no longer needs to send them a link. The proxy roles in config.sso.teacherRoles and config.sso.ownerRoles add a person to the list on their first visit; they never change the role of someone already on it. With these roles the proxy decides the list: someone removed in the panel comes back while the proxy still gives the role, so take the role away at the proxy. Only an owner role claims an unclaimed instance. A cookie a shared browser kept from the previous person never outranks the person the proxy vouches for. Students. The name in a room comes from the token, with nothing to type. One person is one participant of the room, from any device. Per-address limits count the person: behind Teleport the whole class has one address, the proxy's. Competitions. Entrants still sign in with their own key, as without the proxy. Teleport. The application in Teleport points at Colloq's Service, not at an Ingress: # teleport-kube-agent, app_service apps: - name: colloq uri: http://colloq-app.colloq.svc:3000 public_addr: colloq.teleport.corp.example The chart values: ingress: enabled: false # people come through Teleport config: publicUrl: https://colloq.teleport.corp.example sso: jwksUrl: https://teleport.corp.example/.well-known/jwks.json issuer: teleport.corp.example # the Teleport cluster name audience: http://colloq-app.colloq.svc:3000 # the app's uri: Teleport writes it into aud teacherRoles: colloq-teacher # optional networkPolicy: ingressController: from: # the Teleport agent's Pods instead of ingress-nginx - namespaceSelector: matchLabels: {kubernetes.io/metadata.name: teleport} podSelector: matchLabels: {app: teleport-kube-agent} app: egress: - cidrs: [10.20.0.15/32] # the Teleport proxy, for the keys ports: [443] Instead of fetching the keys over the network, put the JWKS document in config.sso.jwks : the chart mounts it from the ConfigMap colloq-sso-jwks and no egress is needed, but a key rotation in Teleport needs a new value. The header is Teleport-Jwt-Assertion by default; config.sso.header sets another. Authorization won't do: the browser sends the room token in it, and a proxy writing its own there would break files, history and the Oracle. oauth2-proxy with --pass-access-token passes the token in X-Forwarded-Access-Token — fine when the IdP issues access tokens as JWTs (Keycloak does); the audience is then whatever the IdP writes into the access token. Token fields. By default: who it is — sub or username ; the name — name , traits.name , traits.full_name , traits.display_name , username ; the email — email , traits.email , traits.mail , username ; roles — roles , groups , traits.groups . config.sso.claims.* sets other paths, and the first non-empty one wins. If the token carries only a login, config.sso.emailDomain turns it into the address teachers are recognised by. A line at start says where the keys come from and whom Colloq adds; a refused token is logged once a minute with the reason. Students without an account. Only someone with an account gets through Teleport. If students have none, keep the ordinary way in for them: an Ingress to the same Service at a public address and the class links, with config.publicUrl set to that public address, which links are built from. Teachers then open both the panel and the rooms at the Teleport address — the panel still builds the links for students from the public one: the token is checked the same way wherever the request came from, and a page at the Teleport address passes the origin check when the Teleport agent is in config.inbound.trustedProxies (the chart sets private ). What stays with the proxy. A token is a pass until it expires: if Colloq can also be reached around the proxy, a stolen token works until its exp . Keep the token lifetime short, and keep the token header out of access logs."},{"title":"Storage","section":"Colloq in your Kubernetes","url":"kubernetes.html#storage","text":"PVC What it holds Who mounts it Default size colloq-data The SQLite database ( colloq.db with its WAL journal), the link-signing key, the setup token, output images, class page files in page-files/ , competition data, hidden answers and submissions, package sets, database snapshots in snapshots/ the app; competition Pods — their own folders 10 GiB; more for competitions with large data, and about 0.3–0.8 GiB for each 33-class course with class pages colloq-workspace Room files, a / folder each the app; a room's Pod and its personal-notebook Pod — only that room's folder 100 GiB The storage class, size and access mode are set for both volumes at once ( persistence.storageClass , persistence.accessMode ) or for each ( persistence.data.* , persistence.workspace.* ); existingClaim attaches a PVC you already have. With persistence.retain: true , the default, the PVCs survive both helm uninstall and deleting the ArgoCD application. ReadWriteMany: preferred. With RWX on both volumes, room Pods spread over the cluster wherever the scheduler puts them, and the installation's capacity is bounded by the quota rather than by one node. CephFS or NFSv4, where the protocol itself handles file locking, will do; SQLite has a single writer, the app. SQLite's own documentation warns that locking is unreliable on many NFS implementations: if in doubt, keep the database volume on block storage ( persistence.data.accessMode: ReadWriteOnce ), which brings back running every Pod with a volume on one node. ReadWriteOnce: one node. This is the chart's default. An RWO volume attaches to one node, so everything that mounts Colloq's volumes must run there: if at least one of the volumes is RWO, the chart sets RUNTIME_COLOCATE_WITH_APP=1 for the broker, and every Pod that mounts a volume — rooms, personal notebooks, submissions, package preparation — gets a required podAffinity to the app's Pod (label colloq.dev/role: app , topology kubernetes.io/hostname ). The app's own Pod prefers the node where rooms already run: after an upgrade the new Pod needs exactly the node the volume is attached to. The broker mounts no volume and can run anywhere. What follows: the whole installation's capacity is one node's free resources; a room beyond them doesn't start, and after 20 seconds of waiting the teacher sees what was short — memory, cores or a GPU; maintenance on that node is a break for all of Colloq. Pick a node with headroom; see Resources and quotas . File ownership. Every Pod writes as uid and gid 1000. The app's Pod sets fsGroup: 1000 , and a fresh block volume becomes writable for group 1000 by itself. For RWX volumes the CSI driver doesn't apply fsGroup by default ( fsGroupPolicy: ReadWriteOnceWithFSType covers only RWO volumes with a filesystem type): the root of such a volume must be writable by uid 1000 on the storage side. No per-room disk quota. The colloq-workspace volume is shared: a student who fills it stops file saving for everyone. Watch PVC usage — the kubelet metrics kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes , or colloq_disk_free_bytes from /metrics — with an alert at 80 %. If the StorageClass allows volume expansion, the PVCs can grow."},{"title":"Images and the registry mirror","section":"Colloq in your Kubernetes","url":"kubernetes.html#images","text":"Image What it is ghcr.io/colloq-edu/colloq-app:vX.Y.Z The app. ghcr.io/colloq-edu/colloq-runtime:vX.Y.Z The broker; also the submissions' result exporter and the package proxy. ghcr.io/colloq-edu/colloq-kernel:vX.Y.Z-base , …-kaggle-base Room and submission kernels: the base and kaggle-base environments, CPU only. Every image is built for amd64 only. A release and its chart share one version number. In the published chart the defaults carry the digest of every image of the release: the app's and the broker's in image.app.digest and image.runtime.digest , the kernels' in the environment catalog catalog.environments . The same values come as a file of their own, values-X.Y.Z.yaml , attached to the release on GitHub: handy for copying the images into a mirror and for diffing releases in a GitOps repository. The chart in the source tree carries no digests and refuses to render without them. The environment chosen for a class is a catalog entry: a name, an image by digest, a GPU flag. An environment of your own — other libraries, or a GPU — is built outside the cluster from the repository's kernel/ directory with KERNEL_ENV= , pushed to your registry and added to catalog.environments : on Kubernetes the panel doesn't build environments. The base-gpu image (about 9 GB of CUDA wheels) is not published; build a GPU environment the same way. The registry mirror. global.imageRegistry replaces the registry in every image reference — the app's, the broker's, the catalog's kernels, the competition exporter and package proxy — and keeps the path and the digest. The value may carry a path too: a Harbor proxy cache of ghcr.io in a project called ghcr works as is, and harbor.corp.example/ghcr gives harbor.corp.example/ghcr/colloq-edu/colloq-app@sha256:… . So the mirror must hold the images under the same paths. Pull secrets go into global.imagePullSecrets : the app's and the broker's Pods get them, and every Pod the broker creates gets the first one. Without internet access. On a machine that can reach ghcr.io : # the release's chart helm pull oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z # every image it refers to, the catalog's kernels included helm template colloq colloq-X.Y.Z.tgz --set ingress.enabled=false \\ | grep -oE '[a-z0-9.-]+/[a-z0-9._/-]+@sha256:[a-f0-9]{64}' | sort -u # each image into the mirror under the same path, with the same digest crane copy ghcr.io/colloq-edu/colloq-app@sha256:… harbor.corp.example/colloq-edu/colloq-app:vX.Y.Z # the chart into your OCI registry helm push colloq-X.Y.Z.tgz oci://harbor.corp.example/charts The chart package colloq-X.Y.Z.tgz is also attached to the release on GitHub, next to values-X.Y.Z.yaml . Any tool that copies the manifest as it is keeps the digest: crane copy , skopeo copy --all --preserve-digests or Harbor replication. Without internet access you lose notebook import from GitHub, the Oracle unless your own model runs on the internal network, and competition entrants' own packages unless you have a PyPI mirror. Everything else works: on Kubernetes students' code gets no internet anyway. What the images are built from. The app and the broker stand on node:22-trixie-slim (Debian 13): the build applies every fix Debian has published so far, and npm, npx, corepack and yarn are removed from the image, since nothing runs them. Kernels stand on python:3.11-slim-bookworm (Debian 12). Every image in the registry carries an SBOM and provenance as BuildKit attestations: docker buildx imagetools inspect ghcr.io/colloq-edu/colloq-app:vX.Y.Z --format '{{json .SBOM}}' . Tools that copy the whole image index ( crane copy , skopeo copy --all ) carry them into the mirror with the image. On release day a scanner such as Trivy finds no critical vulnerability in the app and broker images. Kernels on Debian 12 still have critical findings Debian has not fixed (perl, zlib, sqlite); if your registry refuses to serve images with such findings, tell us: moving the kernels to Debian 13 is the next step."},{"title":"Installation","section":"Colloq in your Kubernetes","url":"kubernetes.html#install","text":"Your team creates the namespace with its Pod Security labels and quota; then Helm or ArgoCD. helm install colloq oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z \\ --namespace colloq --values colloq-values.yaml Values for a bank-like cluster: ReadWriteMany on CephFS, a Harbor proxy cache, secrets from Vault, an internal AI model and a PyPI mirror in Nexus, no internet. global: imageRegistry: harbor.corp.example/ghcr # a proxy cache of ghcr.io imagePullSecrets: [harbor-pull] ingress: host: colloq.corp.example tls: secretName: colloq-tls # the organisation's certificate, type kubernetes.io/tls persistence: accessMode: ReadWriteMany # without RWX keep ReadWriteOnce: everything on one node storageClass: cephfs config: existingSecret: colloq-app-secrets # OPENAI_API_KEY, SESSION_SECRET from Vault timezone: Europe/Moscow institution: Corporate University logFormat: json ai: provider: vllm baseUrl: https://llm.corp.example/v1 model: corp-llm dependencies: indexUrl: https://nexus.corp.example/repository/pypi/simple mirrorEgress: - cidrs: [10.20.0.15/32] # Nexus's addresses: a policy knows addresses, not names ports: [443] publicEgress: false secrets: runtimeToken: existingSecret: colloq-internal # key runtime-token roomSecret: existingSecret: colloq-internal # key room-secret rooms: maxMemory: 16Gi networkPolicy: kubeApiServer: cidrs: [10.0.0.11/32, 10.0.0.12/32, 10.0.0.13/32] ports: [6443] app: egress: - cidrs: [10.30.0.20/32] # the internal AI model ports: [443] metrics: enabled: true existingSecret: colloq-metrics # key metrics-token serviceMonitor: enabled: true Secrets from Vault come through External Secrets. In colloq-app-secrets the keys are the app's variable names ( OPENAI_API_KEY , optionally SESSION_SECRET ); leave config.ai.apiKey and config.sessionSecret empty then. colloq-metrics with the key metrics-token goes the same way as colloq-internal : apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: colloq-internal namespace: colloq spec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: vault target: name: colloq-internal data: - secretKey: runtime-token # openssl rand -hex 32, once remoteRef: {key: education/colloq, property: runtime-token} - secretKey: room-secret remoteRef: {key: education/colloq, property: room-secret} ArgoCD. The chart is an OCI artifact; ArgoCD takes it as a Helm repository with enableOCI , its address without oci:// . apiVersion: v1 kind: Secret metadata: name: corp-charts namespace: argocd labels: argocd.argoproj.io/secret-type: repository stringData: type: helm name: corp-charts url: harbor.corp.example/charts enableOCI: \"true\" --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: colloq namespace: argocd spec: project: education source: repoURL: harbor.corp.example/charts chart: colloq targetRevision: X.Y.Z helm: releaseName: colloq valuesObject: # the values above; or a values file from Git as a second source ingress: host: colloq.corp.example destination: server: https://kubernetes.default.svc namespace: colloq syncPolicy: automated: prune: true selfHeal: true With persistence.retain: true (the default) the chart's PVCs carry argocd.argoproj.io/sync-options: Prune=false,Delete=false , so deleting the application in ArgoCD doesn't delete the data. Room and competition Pods are created by the broker and are not in Git. They deliberately carry no app.kubernetes.io/instance label: ArgoCD before 3.0 tracked its objects by it, and with prune would delete rooms in the middle of a class as superfluous. Your own labels for them go in rooms.podLabels ; don't put app.kubernetes.io/instance there. The owner link. kubectl -n colloq exec deploy/colloq-app -- cat /data/setup-token Open https://colloq.corp.example/admin/t/ , enter your name and email, and you are the owner. While the server has no owner, the same link is printed to the app's log at every start: kubectl -n colloq logs deploy/colloq-app | grep /admin/t/ . Once there is an owner, the token is replaced and never printed again. The link is the key to the server: don't paste it into a group chat, and claim the server right after installing, before the log that carries it spreads through your log pipeline. Add teachers in the panel, under Who can teach : each gets a personal sign-in link. Checks after installing. kubectl -n colloq get deploy,pods — colloq-app and colloq-runtime are Ready. curl -sS https://colloq.corp.example/api/readyz answers {\"ok\":true,\"database\":true,\"workspace\":true} . The full check, from inside the Pod, where error reasons are shown too: kubectl -n colloq exec deploy/colloq-app -- node -e 'fetch(\"http://127.0.0.1:3000/api/health\").then(r => r.text()).then(console.log)' . The answer must have \"ok\":true and \"isolation\":\"broker\" ; the capabilities field says whether competitions are ready. Two rooms from two devices on the students' network: shared editing, a cell run in each, a file upload. kubectl -n colloq get pods -l colloq.dev/role=kernel -o wide shows two Pods and their nodes. Isolation, from a cell in a room: connections to an outside address, to the app's Service and to the node's address must be refused. import socket for host, port in [(\"1.1.1.1\", 443), (\"colloq-app\", 3000), (\"\", 10250)]: try: socket.create_connection((host, port), 3).close() print(host, port, \"OPEN: check the network policies\") except OSError as error: print(host, port, \"closed:\", error) If you use competitions: a trial submission through to the end; leaderboard rows should appear by themselves, with no page reload. Before the first group: a restore from a backup into a separate namespace."},{"title":"Chart values","section":"Colloq in your Kubernetes","url":"kubernetes.html#values","text":"The main values and the variables they set. The full list, each value with an explanation, is the chart's values.yaml : helm show values oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z . Other app settings from the Settings table of the VM guide go into config.extraEnv , or as keys of the Secret named in config.existingSecret . Value Variable and what it sets Default Address and ingress ingress.host The hostname; required while the Ingress is on. — config.publicUrl PUBLIC_URL — the external address links are built from. https:// ingress.className , ingress.annotations The controller class and annotations. nginx , the annotations under Network ingress.tls.secretName , ingress.tls.certManager.* The certificate's Secret, or a cert-manager issuer. colloq-tls config.inbound.trustedProxies TRUSTED_PROXIES — whose X-Forwarded-For to believe. private config.inbound.sharedAddresses SHARED_ADDRESSES — NAT and VPN addresses that per-address caps don't apply to. empty config.inbound.hsts HSTS — the app's Strict-Transport-Security header. true Images and the catalog global.imageRegistry A registry, optionally with a path, instead of ghcr.io for every image, the kernel catalog included. empty global.imagePullSecrets Secrets for pulling images; broker Pods get the first one ( RUNTIME_IMAGE_PULL_SECRET ). — image.app.digest , image.runtime.digest Digests of the app and broker images. the release's digests catalog.environments , catalog.defaultEnvironment KERNEL_CATALOG_FILE , RUNTIME_CATALOG_FILE — the environment catalog: a name, an image by digest, GPU; several revisions of one name with exactly one current: true . the release's base and kaggle-base commonLabels Labels on every chart object and every broker Pod (part of RUNTIME_POD_LABELS ). — Secrets config.existingSecret An existing Secret whose keys become the app's variables. — secrets.runtimeToken.* , secrets.roomSecret.* The broker token and the room secret: existingSecret and existingSecretKey , or value . generated once by the chart (helm CLI only) Storage persistence.accessMode The volumes' access mode. If either volume is ReadWriteOnce, the broker gets RUNTIME_COLOCATE_WITH_APP=1 : every Pod with a volume runs on the app's node. ReadWriteOnce persistence.storageClass , persistence.data.size , persistence.workspace.size The storage class and volume sizes; existingClaim attaches a PVC you have. the cluster default; 10Gi and 100Gi persistence.retain Keep the PVCs when the release or the ArgoCD application is deleted. true Rooms rooms.memory RUNTIME_KERNEL_MEMORY — a room's memory unless the class sets its own. 4Gi rooms.maxMemory RUNTIME_KERNEL_MEMORY_MAX — a room's memory ceiling. Set it explicitly: without it the broker takes its own node's memory minus 1 GiB. the broker node's memory − 1 GiB rooms.cpu RUNTIME_KERNEL_CPU — a room's cores unless the class sets its own. 2 rooms.ephemeralStorage RUNTIME_KERNEL_EPHEMERAL — space for a room's /tmp and home directory. 2Gi rooms.network COLLOQ_ROOM_NETWORK — with none , rooms get no DNS either. DNS only rooms.nodeSelector , rooms.tolerations RUNTIME_ROOM_NODE_SELECTOR , RUNTIME_ROOM_TOLERATIONS — where room and personal-notebook Pods run; competition Pods don't get them. — rooms.priorityClassName RUNTIME_PRIORITY_CLASS — the priority class of every Pod the broker creates. — rooms.podLabels , rooms.podAnnotations RUNTIME_POD_LABELS , RUNTIME_POD_ANNOTATIONS — labels and annotations on every broker Pod; the chart adds app.kubernetes.io/name , instance and part-of to the labels itself. — runtime.inPlaceResize RUNTIME_IN_PLACE_RESIZE — with false , live rooms aren't resized and the Role has no pods/resize rule. true gpu.runtimeClassName RUNTIME_GPU_RUNTIME_CLASS — the RuntimeClass of GPU rooms; empty means none. nvidia gpu.nodeSelector , gpu.tolerations RUNTIME_GPU_NODE_SELECTOR , RUNTIME_GPU_TOLERATIONS — added for GPU rooms. — Network and outbound networkPolicy.ingressController.from Who is admitted to the app's port 3000. ingress-nginx networkPolicy.dns.to Where cluster DNS is. kube-dns in kube-system networkPolicy.kubeApiServer.* The API servers' addresses and ports for the broker; ciliumEntity for Cilium. any address on 443 and 6443 networkPolicy.app.egress Where the app may go outside: the model, a proxy, GitHub. nowhere networkPolicy.monitoring.from Who scrapes /metrics . Prometheus in monitoring outbound.httpsProxy , outbound.httpProxy , outbound.noProxy HTTPS_PROXY , HTTP_PROXY , NO_PROXY — the app's outbound proxy. Service names ( .svc , single-label names) and private addresses bypass it by themselves. — outbound.extraCa.pem or .existingConfigMap NODE_EXTRA_CA_CERTS — the internal CA's certificates as PEM, for the app and package preparation. — dependencies.indexUrl , dependencies.filesHosts DEPENDENCY_INDEX_URL , DEPENDENCY_FILES_HOSTS — the PyPI mirror for competition entrants' own packages: https, no credentials. public PyPI dependencies.mirrorEgress , dependencies.publicEgress The mirror's addresses and ports for the package proxy; the general rule for public addresses on 443. —; true Classes and the Oracle config.timezone TZ — the time zone: day boundaries for daily quotas, dates. Europe/Moscow config.uiLanguage UI_LANGUAGE — the interface language until the owner chooses one in the panel. ru config.maxUploadMb , config.maxSessionMb MAX_UPLOAD_MB , MAX_SESSION_MB — the limit for one file and for all uploads to a room. 50 and 1024 config.ai.provider , .baseUrl , .model , .apiKey AI_PROVIDER , OPENAI_BASE_URL , OPENAI_MODEL , OPENAI_API_KEY — the Oracle's model; for a model on the internal network, vllm , ollama or custom . The key is better kept in config.existingSecret . openai , https://api.openai.com/v1 , gpt-4o-mini config.extraEnv Other app settings, as NAME: value . — Monitoring, backups, resources metrics.enabled , metrics.serviceMonitor.enabled METRICS_TOKEN — /metrics behind a Bearer token, and a ServiceMonitor that uses the same token. off config.logFormat LOG_FORMAT — with json , one JSON object per line. text config.dbSnapshotHours , config.dbSnapshotKeep DB_SNAPSHOT_HOURS , DB_SNAPSHOT_KEEP — how often to write a consistent database snapshot into /data/snapshots , and how many to keep. 24 h and 7 app.resources , runtime.resources Requests and limits of the app and the broker. see Resources and quotas"},{"title":"Resources and quotas","section":"Colloq in your Kubernetes","url":"kubernetes.html#sizing","text":"The numbers come from the limits in the code and from the same reasoning as for one VM , turned here into Pod requests and limits. The main difference from a VM: a room's Pod reserves its memory and cores in full rather than sharing them with its neighbours. Pod How many CPU: request / limit Memory: request / limit Ephemeral storage App 1 250m / 2 512Mi / 2Gi — Broker 1 100m / 1 128Mi / 512Mi — Room one per running room 2 / 2 4Gi / 4Gi 2Gi Personal notebooks one per room that uses them the room's, or the class's own number, request equal to limit; about 250m per student is needed the room's, or Memory per class ; 1–1.5 GiB per student is needed 2Gi Submission one per slot the submission's cores + 100m / + 500m the submission's memory + 64Mi / + 256Mi 128Mi / 320Mi Package preparation while it runs 1100m / 1500m , proxy 100m / 500m 2112Mi / 2304Mi , proxy 64Mi / 128Mi up to 320Mi The minimum quota for one class at a time (a room of 2 cores and 4 GiB), in the ResourceQuota keys requests.cpu , requests.memory , limits.cpu , limits.memory and pods : Scenario CPU requests Memory requests CPU limits Memory limits Pods (a) 30 students, one shared notebook 2.35 4.6 GiB 5 6.5 GiB 3 (b) 30 students, each with a personal notebook (1.5 GiB and 0.25 core per student) 9.85 49.6 GiB 12.5 51.5 GiB 4 (c) 30 students and a competition, 7 slots of 2 cores and 2 GiB 17.05 19.1 GiB 22.5 22.25 GiB 10 (e) 100 people in a lecture room 2.35 4.6 GiB 5 6.5 GiB 3 A room keeps its Pod while anyone is in it and for 2 hours after everyone has left, so back-to-back classes overlap: for each further class within two hours, add another room. Competition slots are one queue for the whole installation. A quota for scenario (c) with headroom for the next class's room and for package preparation: apiVersion: v1 kind: ResourceQuota metadata: name: colloq namespace: colloq spec: hard: requests.cpu: \"22\" requests.memory: 28Gi limits.cpu: \"30\" limits.memory: 32Gi requests.ephemeral-storage: 10Gi pods: \"25\" services: \"100\" persistentvolumeclaims: \"2\" requests.storage: 150Gi The API won't create a Pod over the quota, and the room or the submission gets a refusal. Keep the quota above the sum and the slot count within it: Executors at once in the Resources tab, or COMPETITION_SLOTS in config.extraEnv . The automatic count works from the cores and memory of the node the app runs on and knows nothing about your quota. Services pile up: every room deleted for good leaves a headless Service without a ClusterIP, so give the Service count limit headroom. A LimitRange with a per-container maximum must admit a room (2 cores and 4 GiB; a GPU room with its memory) and the largest submission you allow in competitions: 2 cores and 4 GiB by default. With ReadWriteOnce the whole quota must fit on one node: for scenario (c), a node with at least 24 vCPU and 32 GiB free; for (b), at least 16 vCPU and 64 GiB. The app, measured (30 Sep 2026, a 21 vCPU / 49 GB VM, nginx in front, make load ): 500 people joined one room within a minute with no refusals, a median join taking 7 ms. The app process held 243 MB of memory and 20 % of one core at rest, and half a core with twenty people typing at once; an edit reached everyone else within 0.17 s (p95). At 300 people: 198 MB and the same shares of a core. This was measured on a VM, not in a cluster, but the process is the same: the chart's request of 512 MiB is twice the measured figure, and the limits of 2 cores and 2 GiB leave plenty of headroom."},{"title":"Operations","section":"Colloq in your Kubernetes","url":"kubernetes.html#operations","text":"Upgrades. helm upgrade colloq oci://ghcr.io/colloq-edu/charts/colloq --version X.Y.Z -n colloq -f colloq-values.yaml , or a new targetRevision in ArgoCD. The app and the broker update with the Recreate strategy: the old Pod stops before the new one starts, because SQLite has one writer. The break lasts seconds to a minute; a start with a database migration is given up to 10 minutes. Open rooms reconnect by themselves. Room Pods survive the upgrade: the broker leaves them alone when it stops, the new one adopts them, and Python variables stay. Placement settings — rooms.nodeSelector , rooms.tolerations , the priority class, the storage mode — go to a room's next Pod, while a running one stays where it is. Still, upgrade between classes. Kernel revisions. A room stays on the kernel image revision it started with and runs only that one. On helm upgrade the chart keeps the previous release's revisions in the catalog with current: false by itself (at most catalog.maxRetainedPerEnvironment , three by default, per environment): it reads them from the live colloq-catalog ConfigMap, and rooms started earlier go on with their Python. ArgoCD renders without cluster access and cannot see the previous catalog: there a room whose revision is gone moves to its environment's current revision on its next start, and the app logs a line about it. To keep previous revisions under ArgoCD too, list them in catalog.environments with current: false . Keep the older revisions' images in the mirror while rooms use them. The data version. If a new release raises the database schema version, the app first writes a snapshot, /data/snapshots/pre-schema--to--