diff --git a/CHANGELOG.md b/CHANGELOG.md index 55dea65..1aa25f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,19 @@ reads it dynamically. On every release: bump `__version__` **and** add a section ## [Unreleased] +### Added +- **Linux servers without Proxmox as shutdown targets** (host type `linux`): GPU nodes, a + NAS or any Linux machine on the same UPS is shut down over SSH with the appliance's own + ed25519 key, generated on first use in `/var/lib/pve-usv/ssh/`. A setup script, offered + for download on the host card with the public key filled in, creates a password-less + account, locks the key to a forced command (`restrict`, root-owned `authorized_keys`) + and grants one sudo rule for one root-owned power-off script, which runs the + pre-shutdown hooks in `/etc/pve-ups/pre-shutdown.d/` first. "Test" checks the login and + that sudo rule and records the server's host key, which is pinned from then on. + Thresholds, order, dry-run, the self-test and webhooks apply unchanged. New settings per + host: `ssh_host`, `ssh_port` (22), `ssh_user` (`upsshutdown`), `ssh_host_key` (empty); + existing configs are unaffected. New dependency: `asyncssh`. + ## [4.4.0] - 2026-11-03 A feature release for clusters spread over two rooms: a room that loses power alone is diff --git a/README.de.md b/README.de.md index 6d5d450..4f4ec3d 100644 --- a/README.de.md +++ b/README.de.md @@ -8,7 +8,7 @@ Konfigurationsdateien.** PVE-UPS überwacht eine oder mehrere USVs — **mit SNMP-Netzwerkkarte (Standard RFC 1628 oder Hersteller-MIB: APC PowerNet, CyberPower)** oder **über einen NUT-Server**, worüber USB- und seriell angeschlossene USVs gelesen werden — und fährt bei Stromausfall einen oder mehrere -**Proxmox-VE-Hosts** (auf Wunsch auch einen **Proxmox Backup Server**) geordnet herunter. +**Proxmox-VE-Hosts** (auf Wunsch auch einen **Proxmox Backup Server** oder **einfache Linux-Server** per SSH) geordnet herunter. Der moderne Ersatz für herstellergebundene Appliances wie APC PowerChute Network Shutdown. Die komplette Einrichtung läuft über einen **Web-Wizard**; Monitoring gibt es als **REST/JSON**. @@ -278,14 +278,40 @@ unterscheiden sich von Proxmox VE und erklären die Befehle oben: > Management-Segment erreichbar) oder ihn kurze Ausfälle auf seiner eigenen USV-Laufzeit > aussitzen lassen. +## Linux-Server ohne Proxmox (SSH) + +Auch GPU-Knoten, ein NAS oder jede andere Linux-Maschine an derselben USV lässt sich +herunterfahren: Host-Typ **Linux-Server (SSH)**. Eine API gibt es dort nicht; die Appliance +meldet sich per SSH mit **ihrem eigenen ed25519-Schlüssel** an (beim ersten Bedarf in +`/var/lib/pve-usv/ssh/` erzeugt). Schwellen, Reihenfolge, Dry-Run, täglicher Selbsttest und +Webhooks funktionieren genau wie bei einem Proxmox-Knoten. + +1. Host-Karte anlegen, **Linux-Server (SSH)** wählen, Adresse eintragen. +2. **Einrichtungsskript herunterladen** — es enthält bereits den öffentlichen Schlüssel + dieser Appliance. +3. Einmal als root auf dem Server ausführen: `sh pve-ups-linux-setup.sh` (optional + `--from `). +4. **Testen**: prüft Anmeldung und sudo-Regel und übernimmt den Host-Schlüssel des Servers, + der beim Speichern der Karte festgeschrieben wird. + +Der Schlüssel ist nicht mehr als ein Ausschalter: Er liegt in einer root-eigenen +`authorized_keys`-Datei mit `restrict,command="…"` (keine Shell, kein PTY, kein Forwarding), +und das Forced Command versteht nur `pve-ups test` und `pve-ups shutdown`. Die einzige +sudo-Regel des Kontos erlaubt ein root-eigenes Skript, das die **Pre-Shutdown-Hooks** in +`/etc/pve-ups/pre-shutdown.d/` ausführt (Container stoppen, Slurm drainen, Checkpoint eines +Trainingsjobs — je 120 s, Fehler halten den Shutdown nie auf) und dann ausschaltet. +`--uninstall` entfernt alles. Tipp: GPU-Knoten die niedrigste Reihenfolge-Nummer geben — sie +ziehen die meiste Leistung, und gehen sie zuerst, gewinnen die übrigen Hosts Akkulaufzeit. + ## Funktionen - **Mehrere USVs** pro Instanz mit Host↔USV-Zuordnung und Logik pro Host (**UND** = redundante Netzteile, **ODER** = aufgeteilte Last), inkl. Live-Schaubild. -- **Zwei Ziel-Typen**, in einer Instanz mischbar: **Proxmox-VE**-Knoten und ein **Proxmox - Backup Server** — je Host-Eintrag wählbar, jeweils mit eigenem Token-Schema und eigener - Rechteprüfung (vor dem Anbinden eines PBS in Produktion aber bitte die Warnung oben - lesen). +- **Drei Ziel-Typen**, in einer Instanz mischbar: **Proxmox-VE**-Knoten, ein **Proxmox + Backup Server** — jeweils mit eigenem Token-Schema und eigener Rechteprüfung (vor dem + Anbinden eines PBS in Produktion aber bitte die Warnung oben lesen) — und **einfache + Linux-Server** per SSH mit einem auf ein Forced Command festgelegten Schlüssel, inklusive + Pre-Shutdown-Hooks. - **Zwei USV-Quellen**, in einer Instanz frei mischbar: - **SNMP v1/v2c und v3** (authPriv), nur lesend. Liest die Standard-RFC-1628-UPS-MIB oder eine **Hersteller-MIB** — derzeit **APC PowerNet**, womit APC-Karten @@ -477,7 +503,7 @@ PVE_USV_CONFIG=./dev-config.yaml PVE_USV_DB=./dev-events.db python -m app.main heruntergefahren, und die Vorbereitung wartet auf den letzten Raum. Diese Vorbereitung ist in 4.0.0 **Beta**: Der Mechanismus überprüft jeden Schritt und wählt im Zweifel die sichere Variante, hat bisher aber wenige echte Cluster gesehen. -- Shutdown-Ziele sind **Proxmox VE und Proxmox Backup Server**. Proxmox Mail Gateway und +- Shutdown-Ziele sind **Proxmox VE, Proxmox Backup Server und Linux per SSH**. Proxmox Mail Gateway und Datacenter Manager sind nicht umgesetzt: beide sprechen ihr eigenes Token-Schema, und ungetesteter Support wäre schlechter als keiner. - Liest die Standard-RFC-1628-UPS-MIB, die MIBs von APC PowerNet und CyberPower oder die diff --git a/README.md b/README.md index cd8569c..2a01f76 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ and no config files.** PVE-UPS monitors one or more UPS devices — **with an SNMP network card (standard RFC 1628 or a vendor MIB: APC PowerNet, CyberPower)** or **through a NUT server**, which is how USB and serial UPS devices are read — and, on a power outage, shuts down one or more **Proxmox VE -hosts** (and, if you want, a **Proxmox Backup Server**) in an orderly fashion. The modern +hosts** (and, if you want, a **Proxmox Backup Server** or **plain Linux servers** over SSH) in an orderly fashion. The modern replacement for vendor-locked appliances such as APC PowerChute Network Shutdown. Everything is configured through a **web wizard**; monitoring is available as **REST/JSON**. @@ -265,13 +265,37 @@ from Proxmox VE and explain the commands above: > instance, reachable only from a separate management segment), or let it ride out short > outages on its own UPS runtime. +## Linux servers without Proxmox (SSH) + +GPU nodes, a NAS or any other Linux machine on the same UPS can be shut down too: host +type **Linux server (SSH)**. Such a machine has no API, so the appliance logs in over SSH +with **its own ed25519 key** (generated on first use in `/var/lib/pve-usv/ssh/`). Thresholds, +order, dry-run, the daily self-test and webhooks work exactly as for a Proxmox node. + +1. Add a host card, choose **Linux server (SSH)**, enter the address. +2. Click **Download setup script** — it already carries this appliance's public key. +3. Run it once as root on the server: `sh pve-ups-linux-setup.sh` (optionally + `--from `). +4. Press **Test**: it checks the login and the sudo rule and records the server's host key, + which is pinned when the card is saved. + +The key is no more than a power button: it lives in a root-owned `authorized_keys` file +with `restrict,command="…"` (no shell, no PTY, no forwarding), and the forced command +understands only `pve-ups test` and `pve-ups shutdown`. The account's single sudo rule +allows one root-owned script, which runs the **pre-shutdown hooks** in +`/etc/pve-ups/pre-shutdown.d/` (stop containers, drain Slurm, checkpoint a training job — +120 s each, failures never block the shutdown) and then powers off. `--uninstall` removes +everything. Tip: give GPU nodes the lowest order number — they draw the most power, so +taking them down first buys the other hosts battery time. + ## Features - **Multiple UPS devices** per instance with host↔UPS mapping and per-host logic (**AND** = redundant power supplies, **OR** = split load), including a live feed diagram. -- **Two target types**, mixable in one instance: **Proxmox VE** nodes and a **Proxmox - Backup Server** — selected per host entry, each with its own token scheme and privilege - check (but see the warning above before attaching a PBS in production). +- **Three target types**, mixable in one instance: **Proxmox VE** nodes, a **Proxmox + Backup Server** — each with its own token scheme and privilege check (but see the warning + above before attaching a PBS in production) — and **plain Linux servers** over SSH with a + key locked to a forced command, including pre-shutdown hooks. - **Two UPS sources**, freely mixable in one instance: - **SNMP v1/v2c and v3** (authPriv), read-only. Reads the standard RFC 1628 UPS MIB or a **vendor MIB** — currently **APC PowerNet**, which is what makes APC cards work that @@ -446,7 +470,7 @@ PVE_USV_CONFIG=./dev-config.yaml PVE_USV_DB=./dev-events.db python -m app.main down without it and the preparation waits for the last room. That preparation is **Beta** in 4.0.0: the mechanism verifies every step and defaults to the safe choices, but it has seen few real clusters so far. -- Shutdown targets are **Proxmox VE and Proxmox Backup Server**. Proxmox Mail Gateway and +- Shutdown targets are **Proxmox VE, Proxmox Backup Server and Linux over SSH**. Proxmox Mail Gateway and Datacenter Manager are not implemented: each speaks its own token scheme, and shipping untested support would be worse than none. - Reads the standard RFC 1628 UPS MIB, the APC PowerNet and CyberPower MIBs, or a NUT diff --git a/app/config.py b/app/config.py index 56b39fe..ab6f0b6 100644 --- a/app/config.py +++ b/app/config.py @@ -305,6 +305,7 @@ class HostType(str, Enum): pve = "pve" # Proxmox VE node pbs = "pbs" # Proxmox Backup Server + linux = "linux" # any Linux server, shut down over SSH (see app/linuxssh.py) class HostConfig(BaseModel): @@ -359,6 +360,28 @@ def api_node(self) -> str: """The ``{node}`` path segment of the shutdown call.""" return self.name + @property + def address(self) -> str: + """Where the shutdown goes, in a form two entries can be compared on. + + The API URL for the Proxmox products; a target that is not reached over an API + overrides it. duplicate_api_urls() compares this, so a second SSH entry pointing + at the same machine is caught by the same check as a duplicated PVE URL. + """ + return self.api_url + + def missing_settings(self) -> list[str]: + """The settings without which a shutdown cannot reach this target (see + AppConfig.incomplete_host). Per type, because each one is reached differently.""" + missing = [] + if not self.api_url.strip(): + missing.append("no API URL") + if not self.token_id.strip(): + missing.append("no API token ID") + if not self.token_secret.get_secret_value(): + missing.append("no API token secret") + return missing + @classmethod def secret_fields(cls) -> dict[str, str]: """Secret field names -> default value, for the API's masked-secret reconcile.""" @@ -452,13 +475,64 @@ def api_node(self) -> str: return "localhost" +class LinuxHostConfig(HostConfig): + """Any Linux server without a Proxmox API — a GPU box, a NAS, a plain Debian host. + + Shut down over SSH with the appliance's own key (see app/linuxssh.py). The key in the + target's authorized_keys is locked to one forced command, and that command may only + run one root-owned script through sudo, so the appliance can power the machine off + and nothing else: the SSH counterpart of a token that carries only Sys.PowerMgmt. + + No per-host secret: every Linux target trusts the same appliance key, whose private + half never leaves the appliance's state directory. The API fields inherited from + HostConfig stay empty and unused. + """ + + type: Literal["linux"] = "linux" + api_url: str = "" # unused; overridden only so it is no longer required + + ssh_host: str = "" # address or DNS name of the server + ssh_port: int = Field(default=22, ge=1, le=65535) + ssh_user: str = "upsshutdown" # the account the setup script creates + # The server's host key as " ", pinned on the first successful + # test. Empty means "not pinned yet": the connection is then accepted, because the + # only thing a man in the middle could learn from it is the word "shutdown" — but + # once pinned, a different key is refused (the machine was reinstalled, or it is not + # the machine that was configured). + ssh_host_key: str = "" + + @field_validator("ssh_host", "ssh_user", "ssh_host_key", mode="before") + @classmethod + def _strip(cls, value): + return value.strip() if isinstance(value, str) else value + + @property + def address(self) -> str: + return f"ssh://{self.ssh_host}:{self.ssh_port}" if self.ssh_host else "" + + def missing_settings(self) -> list[str]: + missing = [] + if not self.ssh_host: + missing.append("no SSH address") + if not self.ssh_user: + missing.append("no SSH user") + return missing + + @classmethod + def secret_fields(cls) -> dict[str, str]: + return {} + + # Discriminated union of every supported shutdown target. Adding a type means: a model # here, a branch in app/targets.py, an entry in HostType and the matching i18n keys. -ShutdownTarget = Annotated[Union[PveHostConfig, PbsHostConfig], Field(discriminator="type")] +ShutdownTarget = Annotated[ + Union[PveHostConfig, PbsHostConfig, LinuxHostConfig], Field(discriminator="type") +] TARGET_MODELS: dict[str, type[HostConfig]] = { "pve": PveHostConfig, "pbs": PbsHostConfig, + "linux": LinuxHostConfig, } @@ -1065,14 +1139,7 @@ def incomplete_host(host: HostConfig) -> str: """ if not host.enabled: return "" - missing = [] - if not host.api_url.strip(): - missing.append("no API URL") - if not host.token_id.strip(): - missing.append("no API token ID") - if not host.token_secret.get_secret_value(): - missing.append("no API token secret") - return ", ".join(missing) + return ", ".join(host.missing_settings()) def ordered_hosts(self) -> list[HostConfig]: """Enabled hosts in shutdown order; the appliance's own host always last.""" @@ -1130,7 +1197,7 @@ def duplicate_api_urls(self) -> list[str]: seen: dict[str, int] = {} for host in self.hosts: if host.enabled: - key = _norm_api_url(host.api_url) + key = _norm_api_url(host.address) seen[key] = seen.get(key, 0) + 1 return sorted(url for url, count in seen.items() if url and count > 1) @@ -1142,7 +1209,7 @@ def api_url_is_unique(self, host: HostConfig) -> bool: name (see app/proxmox.py). Where a URL serves several entries, PVE's proxying is the only thing that tells them apart, so the name has to stay in the path. """ - return _norm_api_url(host.api_url) not in self.duplicate_api_urls() + return _norm_api_url(host.address) not in self.duplicate_api_urls() def _norm_api_url(url: str) -> str: diff --git a/app/engine.py b/app/engine.py index 09ca091..0fc9188 100644 --- a/app/engine.py +++ b/app/engine.py @@ -2228,7 +2228,7 @@ async def _warn_about_duplicate_api_urls(self) -> None: self._dup_url_warned.add(url) names = [ h.name for h in self.cfg.hosts - if h.enabled and h.api_url.strip().rstrip("/").lower() == url + if h.enabled and h.address.strip().rstrip("/").lower() == url ] await self._hint( "Several hosts share one API URL", diff --git a/app/linuxssh.py b/app/linuxssh.py new file mode 100644 index 0000000..92a6623 --- /dev/null +++ b/app/linuxssh.py @@ -0,0 +1,235 @@ +"""SSH client for plain Linux shutdown targets (no Proxmox API on the machine). + +The appliance holds ONE ed25519 key of its own, generated on first use in its state +directory. On the target, app/remote/linux-target-setup.sh installs that key for a +dedicated account, locked to a forced command that understands exactly two requests: + + pve-ups test -> PVE-UPS-REMOTE 1 ok power=yes|no host= + pve-ups shutdown -> PVE-UPS-REMOTE 1 accepted + +"power=yes" means the account's single sudo rule — run the root-owned power-off script — +is in place: the SSH counterpart of the Sys.PowerMgmt check on a Proxmox token. The +shutdown request detaches the power-off on the target (systemd-run) and answers at once, +so the reply arrives before the network goes away. + +What is deliberately NOT used: the process's ~/.ssh/config, ssh-agent and known_hosts. +The appliance's behaviour must follow from its own configuration, not from whatever an +operator once put in a home directory. Host keys are pinned per entry instead +(LinuxHostConfig.ssh_host_key), on the first successful test. + +Both entry points never raise, like the API clients behind app/targets.py. + +Copyright 2026 Florian Finder +""" + +from __future__ import annotations + +import logging +import os +import socket +import threading +from importlib import resources +from pathlib import Path +from typing import Optional + +import asyncssh + +from . import db +from .config import LinuxHostConfig +from .proxmox import CONNECT_TIMEOUT_S, TestResult + +log = logging.getLogger("pve-usv.linuxssh") + +KEY_PATH = Path( + os.environ.get("PVE_USV_SSH_KEY", str(db.DB_PATH.parent / "ssh" / "id_ed25519")) +) +KEY_TYPE = "ssh-ed25519" + +CMD_TEST = "pve-ups test" +CMD_SHUTDOWN = "pve-ups shutdown" +REPLY_PREFIX = "PVE-UPS-REMOTE 1 " + +# Marker in the setup script that the UI download replaces with the public key. +SETUP_SCRIPT_KEY_MARKER = "__PVE_UPS_PUBKEY__" + +_key_lock = threading.Lock() +_key_cache: dict[Path, asyncssh.SSHKey] = {} + + +def appliance_key(path: Optional[Path] = None) -> asyncssh.SSHKey: + """The appliance's private key, generated (0600, in a 0700 directory) on first use. + + Raises OSError when the state directory is not writable — the callers below turn that + into a failed result like any other connection error. + """ + path = Path(path or KEY_PATH) + with _key_lock: + cached = _key_cache.get(path) + if cached is not None: + return cached + if path.exists(): + key = asyncssh.read_private_key(str(path)) + else: + key = asyncssh.generate_private_key(KEY_TYPE, comment=_key_comment()) + path.parent.mkdir(parents=True, exist_ok=True) + os.chmod(path.parent, 0o700) + tmp = path.with_suffix(".tmp") + # O_EXCL + mode at creation: the private half is never world-readable, not + # even for the moment between write and chmod. + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + with os.fdopen(fd, "wb") as fh: + fh.write(key.export_private_key("openssh")) + os.replace(tmp, path) + log.info("Generated the appliance SSH key %s", path) + _key_cache[path] = key + return key + + +def _key_comment() -> str: + try: + return f"pve-ups@{socket.gethostname()}" + except OSError: # pragma: no cover - gethostname essentially never fails + return "pve-ups" + + +def public_key_line(path: Optional[Path] = None) -> str: + """The public key as one authorized_keys line ("ssh-ed25519 AAAA… pve-ups@host").""" + key = appliance_key(path) + return key.export_public_key("openssh").decode().strip() + + +def setup_script(path: Optional[Path] = None) -> str: + """The target-side setup script with this appliance's public key filled in.""" + text = (resources.files("app") / "remote" / "linux-target-setup.sh").read_text() + return text.replace(SETUP_SCRIPT_KEY_MARKER, public_key_line(path), 1) + + +def _key_fields(line: str) -> str: + """" " of a key line — comments are not part of the identity.""" + return " ".join((line or "").split()[:2]) + + +def _fingerprint(line: str) -> str: + try: + return asyncssh.import_public_key(_key_fields(line)).get_fingerprint("sha256") + except (asyncssh.KeyImportError, ValueError): + return "?" + + +def _known_hosts(host: LinuxHostConfig): + """None (accept and report) until a key is pinned; then only that key.""" + if not host.ssh_host_key: + return None + return ([asyncssh.import_public_key(_key_fields(host.ssh_host_key))], [], []) + + +async def _run(host: LinuxHostConfig, command: str, timeout: float) -> tuple[str, str]: + """Run one request on the target; returns (reply after the prefix, server host key).""" + conn = await asyncssh.connect( + host.ssh_host, + port=host.ssh_port, + username=host.ssh_user, + client_keys=[appliance_key()], + known_hosts=_known_hosts(host), + config=None, + agent_path=None, + preferred_auth="publickey", + connect_timeout=min(timeout, CONNECT_TIMEOUT_S), + login_timeout=timeout, + ) + try: + server_key = conn.get_server_host_key() + host_key = ( + _key_fields(server_key.export_public_key("openssh").decode()) if server_key else "" + ) + result = await conn.run(command, check=False, timeout=timeout) + finally: + conn.close() + out = result.stdout if isinstance(result.stdout, str) else (result.stdout or b"").decode() + for line in out.splitlines(): + if line.startswith(REPLY_PREFIX): + return line[len(REPLY_PREFIX):].strip(), host_key + got = (out.strip() or str(result.stderr or "").strip() or "no output")[:160] + raise _NotTheRemoteScript(got) + + +class _NotTheRemoteScript(Exception): + """The server answered, but not with the forced command's reply line.""" + + +def _error_text(host: LinuxHostConfig, exc: BaseException) -> str: + where = f"{host.ssh_user}@{host.ssh_host}:{host.ssh_port}" + if isinstance(exc, asyncssh.HostKeyNotVerifiable): + return ( + f"Host key of {where} does not match the pinned one " + f"({_fingerprint(host.ssh_host_key)}). Reinstalled machine, or not the one " + f"configured? Clear the host key on the card and test again to re-pin it." + ) + if isinstance(exc, asyncssh.PermissionDenied): + return ( + f"{where} refused the appliance key. Run the setup script on the server " + f"(it installs the key for '{host.ssh_user}'), then test again." + ) + if isinstance(exc, _NotTheRemoteScript): + return ( + f"{where} accepted the key but did not answer like the PVE-UPS remote script " + f"(got: {exc}). The key must be installed by the setup script, which locks it " + f"to the forced command." + ) + # Before OSError: asyncssh's own TimeoutError derives from both. + if isinstance(exc, TimeoutError): + return f"No answer from {where} in time" + if isinstance(exc, asyncssh.ConnectionLost): + return f"{where} closed the connection before answering" + if isinstance(exc, OSError): + return f"Cannot connect to {where}: {exc.strerror or exc}" + return f"SSH error with {where}: {exc}" + + +async def test_connection(host: LinuxHostConfig, timeout: float = 10.0) -> TestResult: + """Reach the target, check the forced command and its sudo rule, report the host key.""" + try: + reply, host_key = await _run(host, CMD_TEST, timeout) + except Exception as exc: # noqa: BLE001 - never raises, see module docstring + return TestResult(False, _error_text(host, exc)) + + words = reply.split() + status = words[0] if words else "" + fields = dict(w.split("=", 1) for w in words[1:] if "=" in w) + if status != "ok": + return TestResult(False, f"The remote script answered '{reply}'", host_key=host_key) + + remote_name = fields.get("host", "?") + pinned = bool(host.ssh_host_key) + key_note = ( + "Host key matches the pinned one." + if pinned + else f"Host key {_fingerprint(host_key)} recorded — it is pinned when you save." + ) + if fields.get("power") != "yes": + return TestResult( + True, + f"Connection to '{remote_name}' ok, but the sudo rule for the power-off script " + f"is missing (/etc/sudoers.d/pve-ups). Re-run the setup script. {key_note}", + has_power_mgmt=False, + host_key=host_key, + ) + return TestResult( + True, + f"Connection to '{remote_name}' and power-off permission ok. {key_note}", + has_power_mgmt=True, + host_key=host_key, + ) + + +async def shutdown_node(host: LinuxHostConfig, timeout: float = 60.0) -> tuple[bool, str]: + """Ask the target to power off. Returns (ok, message).""" + try: + reply, _ = await _run(host, CMD_SHUTDOWN, timeout) + except Exception as exc: # noqa: BLE001 + msg = _error_text(host, exc) + log.error("Shutdown of %s failed: %s", host.name, msg) + return False, msg + if reply.split()[:1] == ["accepted"]: + return True, "Shutdown command accepted" + return False, f"The remote script answered '{reply}'" diff --git a/app/main.py b/app/main.py index e259eeb..be5a2b9 100644 --- a/app/main.py +++ b/app/main.py @@ -57,7 +57,7 @@ from .engine import Engine, _hostname # ``cluster`` and ``proxmox`` are PVE-specific on purpose: both back wizard checks # that only exist for a Proxmox VE node. Shutdowns still go through ``targets``. -from . import cluster, history, notify, proxmox, sources, targets +from . import cluster, history, linuxssh, notify, proxmox, sources, targets log = logging.getLogger("pve-usv") @@ -1051,9 +1051,41 @@ async def api_test_host(incoming: dict): # None for non-PVE targets; "readable": false means the API would not name its # nodes, which is explicitly NOT a verdict on the configured name. "node_check": node_check, + # SSH targets: the server's host key, which the card pins after a good test. + "host_key": result.host_key, } +def _ssh_key_or_503() -> str: + try: + return linuxssh.public_key_line() + except OSError as exc: + raise HTTPException( + status_code=503, detail=f"The appliance SSH key cannot be created: {exc}" + ) + + +@app.get("/api/linux/ssh-key", dependencies=[Depends(require_auth)]) +def api_linux_ssh_key(): + """The appliance's public SSH key, generated on first request (see app/linuxssh.py). + + Behind the login although a public key is no secret: the setup script below is what + the card offers, and both belong to configuring the appliance. + """ + return {"public_key": _ssh_key_or_503()} + + +@app.get("/api/linux/setup-script", dependencies=[Depends(require_auth)]) +def api_linux_setup_script(): + """The target-side setup script with this appliance's public key filled in.""" + _ssh_key_or_503() + return Response( + content=linuxssh.setup_script(), + media_type="text/x-shellscript; charset=utf-8", + headers={"Content-Disposition": 'attachment; filename="pve-ups-linux-setup.sh"'}, + ) + + @app.post("/api/test/webhook", dependencies=[Depends(require_auth)]) async def api_test_webhook(incoming: dict): """Send one sample notification with the submitted (still unsaved) webhook settings. diff --git a/app/proxmox.py b/app/proxmox.py index 2ee1373..2c837e2 100644 --- a/app/proxmox.py +++ b/app/proxmox.py @@ -37,6 +37,9 @@ class TestResult: # field rather than part of ``ok`` because the two answer different questions — the # credentials can be perfect while the name points at nothing. node_state: str = "unverified" + # SSH targets only: the server's host key as " ", so the UI can pin + # it after the first successful test. Empty for every API target. + host_key: str = "" @dataclass diff --git a/app/remote/linux-target-setup.sh b/app/remote/linux-target-setup.sh new file mode 100644 index 0000000..f411bc7 --- /dev/null +++ b/app/remote/linux-target-setup.sh @@ -0,0 +1,252 @@ +#!/bin/sh +# PVE-UPS — prepare a plain Linux server to be shut down by the appliance over SSH. +# +# Run once as root on the server (GPU node, NAS, any systemd or SysV Linux). It creates: +# +# * a system user (default "upsshutdown") with a locked password; +# * /usr/local/sbin/pve-ups-remote — the forced command of the appliance's key. Whatever +# the appliance sends, only "pve-ups test" and "pve-ups shutdown" do anything; +# * /usr/local/sbin/pve-ups-poweroff — the one root-owned script that user may run via +# sudo. It runs the hooks in /etc/pve-ups/pre-shutdown.d/ and then powers off; +# * /etc/sudoers.d/pve-ups — exactly that one command, nothing else; +# * the appliance's public key in an authorized_keys file owned by root, restricted to +# the forced command (no shell, no PTY, no forwarding), so the account itself cannot +# loosen its own restrictions. +# +# Re-running it is safe and replaces the key. "--uninstall" removes all of the above. +# +# sh linux-target-setup.sh [--user NAME] [--from CIDR] [--uninstall] ['ssh-ed25519 AAAA…'] +# +# The appliance's web UI (Hosts → type "Linux server (SSH)" → "Setup script") hands out +# this file with its public key already filled in below. +# +# Copyright 2026 Florian Finder +set -eu + +PUBKEY='__PVE_UPS_PUBKEY__' +USER_NAME=upsshutdown +FROM="" +UNINSTALL=0 + +REMOTE=/usr/local/sbin/pve-ups-remote +POWEROFF=/usr/local/sbin/pve-ups-poweroff +SUDOERS=/etc/sudoers.d/pve-ups +CONF_DIR=/etc/pve-ups +HOOK_DIR=$CONF_DIR/pre-shutdown.d +KEY_DIR=$CONF_DIR/ssh + +die() { echo "pve-ups: $*" >&2; exit 1; } + +while [ $# -gt 0 ]; do + case "$1" in + --user) [ $# -ge 2 ] || die "--user needs a value"; USER_NAME=$2; shift 2 ;; + --from) [ $# -ge 2 ] || die "--from needs a value"; FROM=$2; shift 2 ;; + --uninstall) UNINSTALL=1; shift ;; + -h|--help) sed -n '2,22p' "$0"; exit 0 ;; + ssh-*|ecdsa-*) PUBKEY=$1; shift ;; + *) die "unknown argument: $1" ;; + esac +done + +[ "$(id -u)" -eq 0 ] || die "run this as root" +case "$USER_NAME" in *[!a-z0-9_-]*|"") die "invalid user name: $USER_NAME" ;; esac + +if [ "$UNINSTALL" -eq 1 ]; then + rm -f "$REMOTE" "$POWEROFF" "$SUDOERS" "$KEY_DIR/$USER_NAME.authorized_keys" + rmdir "$KEY_DIR" 2>/dev/null || true + if id "$USER_NAME" >/dev/null 2>&1; then userdel "$USER_NAME" 2>/dev/null || true; fi + echo "pve-ups: removed. Hooks in $HOOK_DIR were left in place." + exit 0 +fi + +case "$PUBKEY" in + ssh-*" "*|ecdsa-*" "*) ;; + *) die "no public key given — pass the appliance's key as the last argument" ;; +esac +case "$PUBKEY" in *' +'*|*'"'*) die "the public key must be a single line without quotes" ;; esac +case "$FROM" in *[!0-9a-fA-F.:/,*?]*) die "invalid --from value: $FROM" ;; esac + +# --- sudo is the only privilege boundary; without it there is nothing to delegate to --- +if ! command -v sudo >/dev/null 2>&1; then + echo "pve-ups: installing sudo" + if command -v apt-get >/dev/null 2>&1; then apt-get install -y sudo >/dev/null + elif command -v dnf >/dev/null 2>&1; then dnf install -y sudo >/dev/null + elif command -v yum >/dev/null 2>&1; then yum install -y sudo >/dev/null + elif command -v zypper >/dev/null 2>&1; then zypper --non-interactive install sudo >/dev/null + elif command -v apk >/dev/null 2>&1; then apk add sudo >/dev/null + else die "sudo is missing and no known package manager was found — install sudo first" + fi +fi + +# --- the account: no password, a plain shell (sshd runs the forced command through it) --- +if ! id "$USER_NAME" >/dev/null 2>&1; then + if command -v useradd >/dev/null 2>&1; then + useradd --system --no-create-home --home-dir / --shell /bin/sh "$USER_NAME" + else + adduser -S -D -H -h / -s /bin/sh "$USER_NAME" # BusyBox / Alpine + fi +fi +passwd -l "$USER_NAME" >/dev/null 2>&1 || true + +# --- the forced command ----------------------------------------------------------------- +cat >"$REMOTE" <<'EOF' +#!/bin/sh +# Forced command of the PVE-UPS appliance key (installed by linux-target-setup.sh). +# Answers on one line, "PVE-UPS-REMOTE 1 [...]", which the appliance parses. +POWEROFF=/usr/local/sbin/pve-ups-poweroff +case "${SSH_ORIGINAL_COMMAND:-}" in + "pve-ups test") + if sudo -n -l "$POWEROFF" --schedule >/dev/null 2>&1; then p=yes; else p=no; fi + echo "PVE-UPS-REMOTE 1 ok power=$p host=$(hostname)" + ;; + "pve-ups shutdown") + if sudo -n "$POWEROFF" --schedule; then + echo "PVE-UPS-REMOTE 1 accepted" + else + echo "PVE-UPS-REMOTE 1 error sudo-failed" + exit 1 + fi + ;; + *) + echo "PVE-UPS-REMOTE 1 error unknown-command" + exit 2 + ;; +esac +EOF + +# --- the one thing root does for that account ------------------------------------------- +cat >"$POWEROFF" <<'EOF' +#!/bin/sh +# Power-off for the PVE-UPS appliance (installed by linux-target-setup.sh). +# --schedule detach and return at once, so the SSH caller gets its answer before the +# network goes down (this is what the sudo rule allows); +# --run the actual sequence: pre-shutdown hooks, then poweroff. +HOOK_DIR=/etc/pve-ups/pre-shutdown.d +HOOK_TIMEOUT=120 +[ -r /etc/pve-ups/remote.conf ] && . /etc/pve-ups/remote.conf +UNIT=pve-ups-poweroff + +case "${1:-}" in + --schedule) + if command -v systemd-run >/dev/null 2>&1 && [ -d /run/systemd/system ]; then + # A second request while the first is still running is not an error: the machine + # is already on its way down. + systemd-run --quiet --no-block --unit="$UNIT" "$0" --run \ + || systemctl is-active --quiet "$UNIT" \ + || exit 1 + else + nohup "$0" --run >/dev/null 2>&1 & + fi + ;; + --run) + logger -t pve-ups "Shutdown requested by the PVE-UPS appliance (UPS on battery)" || true + if [ -d "$HOOK_DIR" ]; then + for hook in "$HOOK_DIR"/*; do + [ -f "$hook" ] && [ -x "$hook" ] || continue + logger -t pve-ups "pre-shutdown hook: $hook" || true + if command -v timeout >/dev/null 2>&1; then + timeout "$HOOK_TIMEOUT" "$hook" || logger -t pve-ups "hook failed or timed out: $hook" || true + else + "$hook" || logger -t pve-ups "hook failed: $hook" || true + fi + done + fi + if command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ]; then + exec systemctl poweroff + fi + exec shutdown -h now + ;; + *) + echo "usage: $0 --schedule | --run" >&2 + exit 2 + ;; +esac +EOF +chown root:root "$REMOTE" "$POWEROFF" +chmod 0755 "$REMOTE" "$POWEROFF" + +# --- sudoers: that script with that argument, and nothing else --------------------------- +TMP_SUDO=$(mktemp) +cat >"$TMP_SUDO" </dev/null 2>&1 && ! visudo -cqf "$TMP_SUDO"; then + rm -f "$TMP_SUDO" + die "the generated sudoers rule did not validate — nothing was changed" +fi +install -o root -g root -m 0440 "$TMP_SUDO" "$SUDOERS" +rm -f "$TMP_SUDO" + +# --- hooks ---------------------------------------------------------------------------- +mkdir -p "$HOOK_DIR" +if [ ! -e "$HOOK_DIR/README" ]; then + cat >"$HOOK_DIR/README" <<'EOF' +Executable files in this directory run as root, in name order, before the machine +powers off on a PVE-UPS request. Each one gets HOOK_TIMEOUT seconds (default 120, +override in /etc/pve-ups/remote.conf). A failing hook is logged and skipped — it never +stops the shutdown. Keep the total well inside the battery runtime. + +Examples: + 10-docker #!/bin/sh + docker ps -q | xargs -r docker stop -t 60 + 20-slurm #!/bin/sh + scontrol update nodename="$(hostname -s)" state=drain reason="UPS on battery" + 30-checkpoint #!/bin/sh + pkill -USR1 -f train.py; sleep 60 +EOF +fi + +# --- the key: root-owned, so the account cannot widen its own restrictions ----------------- +OPTS="restrict,command=\"$REMOTE\"" +[ -n "$FROM" ] && OPTS="$OPTS,from=\"$FROM\"" +mkdir -p "$KEY_DIR" +chown root:root "$CONF_DIR" "$KEY_DIR" +chmod 0755 "$CONF_DIR" "$KEY_DIR" +printf '%s %s\n' "$OPTS" "$PUBKEY" >"$KEY_DIR/$USER_NAME.authorized_keys" +chown root:root "$KEY_DIR/$USER_NAME.authorized_keys" +chmod 0644 "$KEY_DIR/$USER_NAME.authorized_keys" + +# sshd reads that file through a drop-in, for this one user only. Older sshd without +# Include support gets the key in the user's home instead. +SSHD_DROPIN_DIR=/etc/ssh/sshd_config.d +if [ -d "$SSHD_DROPIN_DIR" ] && grep -qiE '^\s*Include\s+/etc/ssh/sshd_config.d' /etc/ssh/sshd_config 2>/dev/null; then + cat >"$SSHD_DROPIN_DIR/50-pve-ups.conf" </dev/null || true + cp "$KEY_DIR/$USER_NAME.authorized_keys" "$HOME_DIR/.ssh/authorized_keys" + chown -R root:root "$HOME_DIR" + chmod 0755 "$HOME_DIR" "$HOME_DIR/.ssh" + chmod 0644 "$HOME_DIR/.ssh/authorized_keys" +fi + +if command -v sshd >/dev/null 2>&1; then + sshd -t || die "sshd rejects its configuration now — check $SSHD_DROPIN_DIR/50-pve-ups.conf" + # AllowUsers / AllowGroups silently lock the account out; say so instead. + if sshd -T 2>/dev/null | grep -qiE '^(allowusers|allowgroups) '; then + echo "pve-ups: WARNING: sshd uses AllowUsers/AllowGroups — make sure '$USER_NAME' is allowed." + fi +fi +for svc in ssh sshd; do + if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet "$svc" 2>/dev/null; then + systemctl reload "$svc" && break + fi +done + +echo "pve-ups: done. User '$USER_NAME' may now power this machine off from the appliance." +echo "pve-ups: press 'Test' on the host card in the appliance to verify and pin the host key." diff --git a/app/targets.py b/app/targets.py index 5fb489d..4019a33 100644 --- a/app/targets.py +++ b/app/targets.py @@ -17,8 +17,8 @@ import asyncio -from . import pbs, proxmox -from .config import HostConfig, PbsHostConfig, PveHostConfig +from . import linuxssh, pbs, proxmox +from .config import HostConfig, LinuxHostConfig, PbsHostConfig, PveHostConfig from .proxmox import NodeVerdict, TestResult # Headroom over the caller's timeout: the client's own httpx timeout should be what @@ -38,10 +38,13 @@ async def shutdown( ``use_localhost`` only means anything to PVE: it addresses the node behind the entry's API URL directly instead of by its configured name (see proxmox.shutdown_node). PBS - ignores the segment entirely and always has, so the flag never reaches it. + ignores the segment entirely and always has, and an SSH target has no node path at + all, so the flag never reaches either. """ if isinstance(host, PbsHostConfig): call = pbs.shutdown_node(host, timeout) + elif isinstance(host, LinuxHostConfig): + call = linuxssh.shutdown_node(host, timeout) elif isinstance(host, PveHostConfig): call = proxmox.shutdown_node(host, timeout, use_localhost=use_localhost) else: @@ -59,8 +62,8 @@ async def verify_node(host: HostConfig, timeout: float = 10.0) -> NodeVerdict: """Check a target's configured node name against its API. Never raises, always bounded. Only Proxmox VE has a node name that can be wrong. PBS ignores the path segment - entirely (see PbsHostConfig.api_node), so there is nothing to verify and nothing to - report — "unverified" is the honest answer, not a defect. + entirely (see PbsHostConfig.api_node) and a Linux target is reached by address, so + there is nothing to verify and nothing to report — "unverified" is the honest answer, not a defect. """ if not isinstance(host, PveHostConfig): return NodeVerdict(state="unverified") @@ -79,6 +82,8 @@ async def test_connection(host: HostConfig, timeout: float = 10.0) -> TestResult """Verify token + power-management privilege. Never raises, always bounded.""" if isinstance(host, PbsHostConfig): call = pbs.test_connection(host, timeout) + elif isinstance(host, LinuxHostConfig): + call = linuxssh.test_connection(host, timeout) elif isinstance(host, PveHostConfig): call = proxmox.test_connection(host, timeout) else: diff --git a/app/web/app.js b/app/web/app.js index 2d8ba30..783a8ff 100644 --- a/app/web/app.js +++ b/app/web/app.js @@ -573,7 +573,7 @@ const SNMP_MIBS = [["auto", t("mib.auto")], ["rfc1628", t("mib.rfc1628")], ["apc ["cyberpower", t("mib.cyberpower")]]; const DEFAULT_PORTS = { snmp: 161, nut: 3493 }; // Must match config.HostType; tests/test_i18n.py keeps the labels in both dictionaries. -const HOST_TYPES = [["pve", t("htype.pve")], ["pbs", t("htype.pbs")]]; +const HOST_TYPES = [["pve", t("htype.pve")], ["pbs", t("htype.pbs")], ["linux", t("htype.linux")]]; const HOST_DEFAULT_PORTS = { pve: 8006, pbs: 8007 }; const TRISTATE = [["", t("tristate.global")], ["on", t("tristate.on")], ["off", t("tristate.off")]]; // Order and values mirror config.WebhookFormat / WebhookLevel (tests/test_i18n.py checks it). @@ -886,7 +886,7 @@ function formSites() { const sites = {}; const unsited = []; document.querySelectorAll("#hostRows .host-cfg").forEach((el) => { - if (el.querySelector(".h_type").value === "pbs") return; + if (el.querySelector(".h_type").value !== "pve") return; if (!el.querySelector(".h_enabled").checked || !el.querySelector(".h_cluster").checked) return; const name = el.querySelector(".h_name").value.trim() || t("host.newName"); const site = el.querySelector(".h_site").value.trim(); @@ -1158,12 +1158,30 @@ function addHostRow(h, isNew, open) {
- +
-
+
+ +

@@ -1174,7 +1192,7 @@ function addHostRow(h, isNew, open) {
- + @@ -1258,7 +1276,7 @@ function addHostRow(h, isNew, open) { // Cluster membership belongs on the collapsed card for the same reason the ★ does: // it changes what happens at shutdown and should not need unfolding to be seen. The // discovered name is used when the engine has one, the plain word until then. - const inCluster = ty !== "pbs" && el.querySelector(".h_cluster").checked; + const inCluster = ty === "pve" && el.querySelector(".h_cluster").checked; const cname = inCluster ? (CLUSTER_NAMES[el.querySelector(".h_id").value] || "") : ""; const site = inCluster ? el.querySelector(".h_site").value.trim() : ""; el.querySelector(".h_sum_meta").textContent = @@ -1275,27 +1293,35 @@ function addHostRow(h, isNew, open) { const toggleHostType = () => { const ty = el.querySelector(".h_type").value; const pbs = ty === "pbs"; + const linux = ty === "linux"; const lbl = el.querySelector(".h_namelbl"); - lbl.title = t(pbs ? "host.nameTitle" : "host.nodeTitle"); - el.querySelector(".h_namecap").textContent = t(pbs ? "host.name" : "host.node"); - el.querySelector(".h_name").placeholder = pbs ? "Backup-Server" : "pve01"; + // Only a PVE node name is load-bearing; for the other two it is a free label. + lbl.title = t(ty === "pve" ? "host.nodeTitle" : "host.nameTitle"); + el.querySelector(".h_namecap").textContent = t(ty === "pve" ? "host.node" : "host.name"); + el.querySelector(".h_name").placeholder = pbs ? "Backup-Server" : (linux ? "gpu01" : "pve01"); + // API fields vs SSH fields. Hidden, not cleared: switching the type back finds them. + el.querySelector(".h_urllbl").hidden = linux; + el.querySelector(".h_apirow").hidden = linux; + el.querySelector(".h_verifylbl").hidden = linux; + el.querySelector(".h_sshbox").hidden = !linux; el.querySelector(".h_url").placeholder = pbs ? "https://10.0.0.20:8007" : "https://10.0.0.10:8006"; el.querySelector(".h_token_id").placeholder = pbs ? "ups@pbs!shutdown" : "ups@pve!shutdown"; el.querySelector(".h_hinttext").textContent = t("htype." + ty + "Help"); const doc = el.querySelector(".h_hintdoc"); doc.textContent = t("host.hintDoc"); - doc.dataset.manual = pbs ? "token-pbs" : "token-pve"; + doc.dataset.manual = pbs ? "token-pbs" : (linux ? "linux-ssh" : "token-pve"); applyTranslations(el); // rewrites the doc link to the current language's manual syncFlags(); }; // Which of the flags can apply at all. Order matters here: the "this host" rule may // clear the tick, and the order field has to come back in the same pass. const syncFlags = () => { - const pbs = el.querySelector(".h_type").value === "pbs"; - // An LXC never runs on a Backup Server, so "this host" cannot apply there. In a - // Docker deployment the container may well sit on the PBS — and then the mark - // matters, or the appliance kills itself midway through the sequence. - const impossible = pbs && deployment !== "docker"; + const ty = el.querySelector(".h_type").value; + const notPve = ty !== "pve"; + // An LXC never runs on a Backup Server or a plain Linux server, so "this host" cannot + // apply there. In a Docker deployment the container may well sit on one — and then + // the mark matters, or the appliance kills itself midway through the sequence. + const impossible = notPve && deployment !== "docker"; el.querySelector(".h_thislbl").hidden = impossible; // Clear it as well: hostFromRow() still reads the checkbox, and a hidden tick // would keep being submitted. @@ -1304,10 +1330,10 @@ function addHostRow(h, isNew, open) { // host is last whatever it says. Hide the field rather than let it suggest an // effect it does not have. The value is kept, so unticking restores it. el.querySelector(".h_orderlbl").hidden = el.querySelector(".h_this").checked; - // A Backup Server is never part of a PVE cluster, so the whole group is PVE-only. + // Only a PVE node is part of a PVE cluster, so the whole group is PVE-only. // Unlike "this host" the values are NOT cleared: switching a card back to PVE should // find the cluster settings as they were. - el.querySelector(".clusterbox").hidden = pbs; + el.querySelector(".clusterbox").hidden = notPve; // The sub-switches only mean anything once the host is marked as a cluster member. el.querySelector(".h_clusteropts").hidden = !el.querySelector(".h_cluster").checked; // The Ceph note describes what the tick does; showing it while the tick is off would @@ -1348,6 +1374,20 @@ function addHostRow(h, isNew, open) { el.querySelector(".h_name").oninput = () => { updSum(); drawConfigTopology(); syncSites(); }; el.querySelector(".h_site").oninput = () => { updSum(); syncSites(); syncClusterThresholds(); }; el.querySelector(".h_url").oninput = syncDuplicateUrls; + el.querySelector(".h_ssh_host").oninput = syncDuplicateUrls; + el.querySelector(".h_ssh_port").oninput = syncDuplicateUrls; + // Forgetting the pinned key is the deliberate way through "host key does not match" + // after a reinstall: the next test records the new one. + el.querySelector(".h_ssh_forget").onclick = () => { el.querySelector(".h_ssh_host_key").value = ""; }; + el.querySelector(".h_ssh_showkey").onclick = async (ev) => { + ev.preventDefault(); + const pre = el.querySelector(".h_ssh_pubkey"); + if (!pre.hidden) { pre.hidden = true; return; } + try { + pre.textContent = (await api("/api/linux/ssh-key")).public_key; + } catch (e) { pre.textContent = "✗ " + e.message; } + pre.hidden = false; + }; el.querySelector(".h_this").onchange = () => { syncFlags(); drawConfigTopology(); }; el.querySelector(".h_cluster").onchange = () => { syncFlags(); syncClusterThresholds(); }; el.querySelector(".h_cluster_ceph").onchange = () => { syncFlags(); syncClusterThresholds(); }; @@ -1370,8 +1410,10 @@ function addHostRow(h, isNew, open) { const ty = el.querySelector(".h_type").value; const urlEl = el.querySelector(".h_url"); const known = Object.values(HOST_DEFAULT_PORTS).map(String); - urlEl.value = urlEl.value.replace(/:(\d+)\/?$/, (m, port) => - known.includes(port) ? ":" + HOST_DEFAULT_PORTS[ty] : m); + if (HOST_DEFAULT_PORTS[ty]) { + urlEl.value = urlEl.value.replace(/:(\d+)\/?$/, (m, port) => + known.includes(port) ? ":" + HOST_DEFAULT_PORTS[ty] : m); + } toggleHostType(); drawConfigTopology(); syncDuplicateUrls(); @@ -1387,16 +1429,25 @@ $("addHostBtn").onclick = () => { addHostRow({}, true, true); renderHostUpsCheck // addressed by the configured node name, because PVE's proxying is then the only thing // telling the entries apart. Flagged while editing, and again by the self-test for configs // that arrive by backup import and never pass through this form. +// Mirrors HostConfig.address: the API URL, or ssh://host:port for a Linux target. +function hostAddress(tr) { + if (tr.querySelector(".h_type").value === "linux") { + const h = tr.querySelector(".h_ssh_host").value.trim().toLowerCase(); + return h ? `ssh://${h}:${Number(tr.querySelector(".h_ssh_port").value || 22)}` : ""; + } + return tr.querySelector(".h_url").value.trim().replace(/\/+$/, "").toLowerCase(); +} + function syncDuplicateUrls() { const rows = Array.from(document.querySelectorAll("#hostRows .host-cfg")); const count = {}; rows.forEach((tr) => { if (!tr.querySelector(".h_enabled").checked) return; - const url = tr.querySelector(".h_url").value.trim().replace(/\/+$/, "").toLowerCase(); + const url = hostAddress(tr); if (url) count[url] = (count[url] || 0) + 1; }); rows.forEach((tr) => { - const url = tr.querySelector(".h_url").value.trim().replace(/\/+$/, "").toLowerCase(); + const url = hostAddress(tr); const dupe = tr.querySelector(".h_enabled").checked && count[url] > 1; tr.querySelector(".h_dupurl").hidden = !dupe; }); @@ -1474,6 +1525,11 @@ function hostFromRow(tr) { })(), cluster_ha_disarm: tr.querySelector(".h_cluster_ha_disarm").checked, cluster_site: tr.querySelector(".h_site").value.trim(), + // Linux (SSH) target. Sent for every type, like the cluster fields above. + ssh_host: tr.querySelector(".h_ssh_host").value.trim(), + ssh_port: Number(tr.querySelector(".h_ssh_port").value || 22), + ssh_user: tr.querySelector(".h_ssh_user").value.trim(), + ssh_host_key: tr.querySelector(".h_ssh_host_key").value.trim(), }; } @@ -1542,6 +1598,10 @@ async function testHost(el) { try { const r = await api("/api/test/host", "POST", hostFromRow(el)); msg.textContent = (r.ok ? "✓ " : "✗ ") + r.message; + // Pin the host key on the first good test; never overwrite a pinned one silently — + // a mismatch already failed the test, and "Forget" is the deliberate way out. + const hk = el.querySelector(".h_ssh_host_key"); + if (r.ok && r.host_key && !hk.value.trim()) hk.value = r.host_key; applyNodeState(msg, r.node_state); applyNodeCheck(el, msg, r.node_check); const c = r.cluster; @@ -1971,6 +2031,13 @@ function incompleteCards() { return; } if (!el.querySelector(".h_enabled").checked) return; + if (el.querySelector(".h_type").value === "linux") { + // No secret on a Linux card (the appliance key is shared), so the address is all. + if (!el.querySelector(".h_ssh_host").value.trim()) { + bad.push([el, t("save.needHostSsh", { who })]); + } + return; + } if (!el.querySelector(".h_url").value.trim()) { bad.push([el, t("save.needHostUrl", { who })]); } else if (!el.querySelector(".h_token_id").value.trim()) { diff --git a/app/web/handbuch.html b/app/web/handbuch.html index 1935e06..7726747 100644 --- a/app/web/handbuch.html +++ b/app/web/handbuch.html @@ -456,6 +456,55 @@

Proxmox Backup Server

getrennten Management-Segment erreichbar) oder ihn kurze Ausfälle auf seiner eigenen USV-Laufzeit aussitzen lassen.
+

Linux-Server ohne Proxmox (SSH)

+

Der Host-Typ Linux-Server (SSH) fährt jede Linux-Maschine herunter, die an + derselben USV hängt — GPU-Knoten, ein NAS, ein einfacher Debian- oder RHEL-Server. + Eine API gibt es dort nicht; die Appliance meldet sich deshalb per SSH mit + ihrem eigenen Schlüssel an, der beim ersten Bedarf erzeugt und in + /var/lib/pve-usv/ssh/ abgelegt wird. Schwellen, Reihenfolge, Dry-Run, + Selbsttest und Webhooks gelten genau wie für einen Proxmox-Knoten.

+
    +
  1. Host-Karte anlegen, Linux-Server (SSH) wählen, Adresse eintragen.
    2. +
  2. Einrichtungsskript herunterladen klicken. Es enthält bereits den + öffentlichen Schlüssel dieser Appliance.
    3. +
  3. Auf den Server kopieren und dort einmal als root ausführen: + sh pve-ups-linux-setup.sh (optional + --from 10.0.0.50, damit der Schlüssel nur von der Adresse der + Appliance akzeptiert wird).
    4. +
  4. Testen drücken. Der erste erfolgreiche Test übernimmt den + Host-Schlüssel des Servers; mit dem Speichern wird er festgeschrieben.
    5. +
+

Was das Skript einrichtet — und warum es nicht mehr als ein Ausschalter ist:

+
    +
  • Ein Systemkonto upsshutdown ohne Passwort.
    • +
  • Den Schlüssel in einer root-eigenen Datei, mit + restrict,command="…": keine Shell, kein PTY, kein Forwarding. Egal + was gesendet wird, nur pve-ups test und + pve-ups shutdown bewirken etwas.
    • +
  • Eine sudo-Regel, /etc/sudoers.d/pve-ups, für genau ein + root-eigenes Skript, /usr/local/sbin/pve-ups-poweroff --schedule. + „Testen“ prüft diese Regel — das Gegenstück zur Prüfung auf + Sys.PowerMgmt.
    • +
  • Pre-Shutdown-Hooks: ausführbare Dateien in + /etc/pve-ups/pre-shutdown.d/ laufen als root in Namensreihenfolge vor + dem Ausschalten — Container stoppen, einen Slurm-Knoten drainen, einem + Trainingsjob das Signal zum Checkpoint geben. Jeder bekommt 120 s + (HOOK_TIMEOUT in /etc/pve-ups/remote.conf); ein + fehlschlagender Hook wird protokolliert und übersprungen, er hält den Shutdown nie + auf.
    • +
+

sh pve-ups-linux-setup.sh --uninstall entfernt alles wieder.

+
+ GPU-Knoten: die niedrigste Reihenfolge-Nummer geben. Sie + ziehen die meiste Leistung; gehen sie zuerst, gewinnen die übrigen Hosts + Akkulaufzeit. Die Summe der Hooks deutlich unter der gemeldeten Restlaufzeit + halten. +
+
+ Meldet ein Test „Host key … does not match the pinned one“, wurde der Server + neu installiert — oder es ist nicht die konfigurierte Maschine. Erst nach Prüfung + Schlüssel vergessen drücken, erneut testen und speichern. +
diff --git a/app/web/i18n/de.js b/app/web/i18n/de.js index e4de6a1..9db823b 100644 --- a/app/web/i18n/de.js +++ b/app/web/i18n/de.js @@ -383,6 +383,7 @@ I18N.de = { "save.cardNo": "Karte {n}", "save.needHostName": "Host {who} hat keinen Knotennamen — bitte ausfüllen oder die Karte entfernen. Beim Speichern ginge er samt API-Token verloren.", "save.needHostUrl": "Host {who} hat keine API-URL — bitte ausfüllen, die Karte entfernen oder „aktiv“ abwählen. Ohne sie hat der Shutdown kein Ziel, und das zeigt sich erst im Stromausfall.", + "save.needHostSsh": "Host {who} hat keine SSH-Adresse — bitte ausfüllen, die Karte entfernen oder „aktiv“ abwählen. Ohne sie hat der Shutdown kein Ziel, und das zeigt sich erst im Stromausfall.", "save.needHostToken": "Host {who} hat keine Token-ID — bitte ausfüllen, die Karte entfernen oder „aktiv“ abwählen. Ohne sie wird der Shutdown abgelehnt, und das zeigt sich erst im Stromausfall.", "save.needHostSecret": "Der neue Host {who} hat kein Token-Secret — bitte ausfüllen, die Karte entfernen oder „aktiv“ abwählen. Es würde leer gespeichert und mit 401 abgewiesen, und das zeigt sich erst im Stromausfall.", "save.needUps": "USV {who} hat weder Adresse noch Namen — bitte eines ausfüllen oder die Karte entfernen. Beim Speichern ginge sie samt gespeicherter Passwörter verloren.", @@ -496,6 +497,9 @@ I18N.de = { "htype.pbsShort": "Proxmox BS", "htype.pveHelp": "Braucht ein API-Token mit dem Privileg „Sys.PowerMgmt“ auf /nodes.", "htype.pbsHelp": "Standard-Port 8007. Braucht ein API-Token mit dem Privileg „Sys.PowerManagement“ auf /system/status — auf PBS trägt das nur die Rolle „Admin“, und sowohl der Benutzer als auch das Token brauchen den Eintrag.", + "htype.linux": "Linux-Server (SSH)", + "htype.linuxShort": "Linux", + "htype.linuxHelp": "Beliebiger Linux-Server ohne Proxmox, heruntergefahren per SSH mit dem eigenen Schlüssel dieser Appliance. Das Einrichtungsskript hinterlegt den Schlüssel auf dem Server, festgelegt auf ein Forced Command, das die Maschine nur ausschalten kann (und Ihre Pre-Shutdown-Hooks ausführt).", // dynamic host row (settings) "host.type": "Typ", @@ -511,6 +515,19 @@ I18N.de = { "host.apiurl": "API-URL", "host.apiurlTitle": "Basis-URL der API inkl. Port, z.B. https://10.0.0.10:8006 (Proxmox VE) oder https://10.0.0.20:8007 (Backup Server)", "host.tokenId": "Token-ID", + "host.sshHost": "SSH-Adresse", + "host.sshHostTitle": "IP-Adresse oder DNS-Name des Servers", + "host.sshPort": "Port", + "host.sshPortTitle": "SSH-Port des Servers, meist 22", + "host.sshUser": "Benutzer", + "host.sshUserTitle": "Das Konto, das das Einrichtungsskript auf dem Server anlegt (Standard upsshutdown)", + "host.sshHostKey": "Host-Schlüssel", + "host.sshHostKeyTitle": "Der SSH-Host-Schlüssel des Servers. Wird beim ersten erfolgreichen Test übernommen und danach bei jeder Verbindung geprüft.", + "host.sshHostKeyPh": "wird beim ersten Test übernommen", + "host.sshForget": "Schlüssel vergessen", + "host.sshSetup": "Server einmalig einrichten: Skript herunterladen, dort als root ausführen (sh pve-ups-linux-setup.sh), dann „Testen“ drücken.", + "host.sshDownload": "Einrichtungsskript herunterladen", + "host.sshShowKey": "Öffentlichen Schlüssel zeigen", "host.tokenIdTitle": "API-Token-Kennung im Format benutzer@realm!tokenname, z.B. ups@pve!shutdown. Im PVE-Cluster gilt dasselbe Token für jeden Knoten.", "host.tokenSecret": "Token-Secret", "host.tokenSecretTitle": "Geheimer Token-Wert (UUID). Wird sicher gespeichert und nie zurückgegeben.", diff --git a/app/web/i18n/en.js b/app/web/i18n/en.js index 2594027..26e77c4 100644 --- a/app/web/i18n/en.js +++ b/app/web/i18n/en.js @@ -383,6 +383,7 @@ I18N.en = { "save.cardNo": "card {n}", "save.needHostName": "Host {who} has no node name — fill it in or remove the card. Saving would discard it, API token included.", "save.needHostUrl": "Host {who} has no API URL — fill it in, remove the card, or untick “active”. Without it the shutdown has nowhere to go, and that only shows during an outage.", + "save.needHostSsh": "Host {who} has no SSH address — fill it in, remove the card, or untick “active”. Without it the shutdown has nowhere to go, and that only shows during an outage.", "save.needHostToken": "Host {who} has no token ID — fill it in, remove the card, or untick “active”. Without it the shutdown is refused, and that only shows during an outage.", "save.needHostSecret": "New host {who} has no token secret — fill it in, remove the card, or untick “active”. It would be stored empty and rejected with 401, which only shows during an outage.", "save.needUps": "UPS {who} has neither an address nor a name — fill one in or remove the card. Saving would discard it, stored passwords included.", @@ -496,6 +497,9 @@ I18N.en = { "htype.pbsShort": "Proxmox BS", "htype.pveHelp": "Needs an API token with the “Sys.PowerMgmt” privilege on /nodes.", "htype.pbsHelp": "Default port 8007. Needs an API token with the “Sys.PowerManagement” privilege on /system/status — on PBS only the “Admin” role carries it, and both the user and the token need the entry.", + "htype.linux": "Linux server (SSH)", + "htype.linuxShort": "Linux", + "htype.linuxHelp": "Any Linux server without Proxmox, shut down over SSH with this appliance's own key. The setup script installs the key on the server, locked to a forced command that can only power the machine off (and run your pre-shutdown hooks).", // dynamic host row (settings) "host.type": "Type", @@ -511,6 +515,19 @@ I18N.en = { "host.apiurl": "API URL", "host.apiurlTitle": "Base URL of the API including port, e.g. https://10.0.0.10:8006 (Proxmox VE) or https://10.0.0.20:8007 (Backup Server)", "host.tokenId": "Token ID", + "host.sshHost": "SSH address", + "host.sshHostTitle": "IP address or DNS name of the server", + "host.sshPort": "Port", + "host.sshPortTitle": "SSH port of the server, usually 22", + "host.sshUser": "User", + "host.sshUserTitle": "The account the setup script creates on the server (default upsshutdown)", + "host.sshHostKey": "Host key", + "host.sshHostKeyTitle": "The server's SSH host key. Recorded by the first successful test and checked on every connection after that.", + "host.sshHostKeyPh": "recorded by the first test", + "host.sshForget": "Forget key", + "host.sshSetup": "Set up the server once: download the script, run it there as root (sh pve-ups-linux-setup.sh), then press “Test”.", + "host.sshDownload": "Download setup script", + "host.sshShowKey": "Show public key", "host.tokenIdTitle": "API token identifier in the form user@realm!tokenname, e.g. ups@pve!shutdown. In a PVE cluster the same token is valid for every node.", "host.tokenSecret": "Token secret", "host.tokenSecretTitle": "Secret token value (UUID). Stored securely and never returned.", diff --git a/app/web/i18n/it.js b/app/web/i18n/it.js index a4e4777..0d7d289 100644 --- a/app/web/i18n/it.js +++ b/app/web/i18n/it.js @@ -383,6 +383,7 @@ I18N.it = { "save.cardNo": "scheda {n}", "save.needHostName": "L'host {who} non ha nome nodo — compilalo o rimuovi la scheda. Il salvataggio lo scarterebbe, token API incluso.", "save.needHostUrl": "L'host {who} non ha URL API — compilalo, rimuovi la scheda o deseleziona “attivo”. Senza di esso lo spegnimento non ha dove andare, e questo si scopre solo durante un blackout.", + "save.needHostSsh": "L'host {who} non ha indirizzo SSH — compilalo, rimuovi la scheda o deseleziona “attivo”. Senza di esso lo spegnimento non ha dove andare, e questo si scopre solo durante un blackout.", "save.needHostToken": "L'host {who} non ha token ID — compilalo, rimuovi la scheda o deseleziona “attivo”. Senza di esso lo spegnimento viene rifiutato, e questo si scopre solo durante un blackout.", "save.needHostSecret": "Il nuovo host {who} non ha segreto del token — compilalo, rimuovi la scheda o deseleziona “attivo”. Verrebbe salvato vuoto e rifiutato con 401, e questo si scopre solo durante un blackout.", "save.needUps": "L'UPS {who} non ha né indirizzo né nome — compilane uno o rimuovi la scheda. Il salvataggio lo scarterebbe, password memorizzate incluse.", @@ -496,6 +497,9 @@ I18N.it = { "htype.pbsShort": "Proxmox BS", "htype.pveHelp": "Richiede un token API con il privilegio “Sys.PowerMgmt” su /nodes.", "htype.pbsHelp": "Porta di default 8007. Richiede un token API con il privilegio “Sys.PowerManagement” su /system/status — su PBS solo il ruolo “Admin” lo possiede, e la voce serve sia all'utente sia al token.", + "htype.linux": "Server Linux (SSH)", + "htype.linuxShort": "Linux", + "htype.linuxHelp": "Qualsiasi server Linux senza Proxmox, spento via SSH con la chiave propria di questa appliance. Lo script di configurazione installa la chiave sul server, vincolata a un forced command che può solo spegnere la macchina (ed eseguire i tuoi hook pre-spegnimento).", // dynamic host row (settings) "host.type": "Tipo", @@ -511,6 +515,19 @@ I18N.it = { "host.apiurl": "URL API", "host.apiurlTitle": "URL base dell'API inclusa la porta, es. https://10.0.0.10:8006 (Proxmox VE) o https://10.0.0.20:8007 (Backup Server)", "host.tokenId": "Token ID", + "host.sshHost": "Indirizzo SSH", + "host.sshHostTitle": "Indirizzo IP o nome DNS del server", + "host.sshPort": "Porta", + "host.sshPortTitle": "Porta SSH del server, di solito 22", + "host.sshUser": "Utente", + "host.sshUserTitle": "L'account che lo script di configurazione crea sul server (default upsshutdown)", + "host.sshHostKey": "Chiave host", + "host.sshHostKeyTitle": "La chiave host SSH del server. Registrata al primo test riuscito e verificata a ogni connessione successiva.", + "host.sshHostKeyPh": "registrata al primo test", + "host.sshForget": "Dimentica chiave", + "host.sshSetup": "Configura il server una volta: scarica lo script, eseguilo lì come root (sh pve-ups-linux-setup.sh), poi premi “Test”.", + "host.sshDownload": "Scarica script di configurazione", + "host.sshShowKey": "Mostra chiave pubblica", "host.tokenIdTitle": "Identificatore del token API nella forma utente@realm!nometoken, es. ups@pve!shutdown. In un cluster PVE lo stesso token è valido per ogni nodo.", "host.tokenSecret": "Segreto del token", "host.tokenSecretTitle": "Valore segreto del token (UUID). Salvato in modo sicuro e mai restituito.", diff --git a/app/web/manual.html b/app/web/manual.html index c8afc43..d04bbb5 100644 --- a/app/web/manual.html +++ b/app/web/manual.html @@ -447,6 +447,50 @@

Proxmox Backup Server

(its own PVE-UPS instance, reachable only from a separate management segment), or let it ride out short outages on its own UPS runtime.
+

Linux servers without Proxmox (SSH)

+

Host type Linux server (SSH) shuts down any Linux machine that is fed by + the same UPS — GPU nodes, a NAS, a plain Debian or RHEL box. There is no API on such + a machine, so the appliance logs in over SSH with its own key, generated on + first use and kept in /var/lib/pve-usv/ssh/. Thresholds, order, + dry-run, the self-test and webhooks apply exactly as for a Proxmox node.

+
    +
  1. Add a host card, pick Linux server (SSH), enter the address.
    2. +
  2. Click Download setup script. It already contains this appliance's + public key.
    3. +
  3. Copy it to the server and run it once as root: + sh pve-ups-linux-setup.sh (optionally + --from 10.0.0.50 to accept the key only from the appliance's + address).
    4. +
  4. Press Test. The first good test records the server's host key; save the + card to pin it.
    5. +
+

What the script sets up, and why it is no more than a power button:

+
    +
  • A system account upsshutdown without a password.
    • +
  • The key, in a root-owned file, with restrict,command="…": no + shell, no PTY, no forwarding. Whatever is sent, only + pve-ups test and pve-ups shutdown do anything.
    • +
  • One sudo rule, /etc/sudoers.d/pve-ups, for one root-owned + script, /usr/local/sbin/pve-ups-poweroff --schedule. “Test” checks + that rule — the counterpart of the Sys.PowerMgmt check.
    • +
  • Pre-shutdown hooks: executable files in + /etc/pve-ups/pre-shutdown.d/ run as root, in name order, before the + power-off — stop containers, drain a Slurm node, signal a training job to write + a checkpoint. Each gets 120 s (HOOK_TIMEOUT in + /etc/pve-ups/remote.conf); a failing hook is logged and skipped, + never blocking the shutdown.
    • +
+

sh pve-ups-linux-setup.sh --uninstall removes everything again.

+
+ GPU nodes: give them the lowest order number. They draw the + most power, so taking them down first buys the remaining hosts battery time. Keep + the hooks' total well inside the runtime the UPS reports. +
+
+ If a test answers “host key does not match the pinned one”, the server was + reinstalled — or it is not the machine that was configured. Only after checking, + press Forget key, test again and save. +
diff --git a/pyproject.toml b/pyproject.toml index 8571001..e3f0082 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,6 +36,7 @@ dependencies = [ "passlib[argon2]>=1.7", # UI password hashing "itsdangerous>=2.1", # signed session cookies "python-multipart>=0.0.9", # multipart form parsing for the update upload + "asyncssh>=2.14", ] [project.optional-dependencies] @@ -68,4 +69,4 @@ packages = ["app"] version = { attr = "app.__version__" } [tool.setuptools.package-data] -app = ["web/*", "web/i18n/*"] +app = ["web/*", "web/i18n/*", "remote/*"] diff --git a/tests/test_linuxssh.py b/tests/test_linuxssh.py new file mode 100644 index 0000000..8552076 --- /dev/null +++ b/tests/test_linuxssh.py @@ -0,0 +1,233 @@ +"""Linux (SSH) shutdown target: model, config round trip and the client against a real +in-process SSH server that plays the forced command from app/remote/linux-target-setup.sh.""" + +from __future__ import annotations + +import asyncio +import os +import stat +import subprocess +from pathlib import Path + +import asyncssh +import pytest + +from app import linuxssh, targets +from app.config import ( + AppConfig, + LinuxHostConfig, + PveHostConfig, + load_config, + save_config, +) + + +@pytest.fixture +def key_path(tmp_path, monkeypatch): + path = tmp_path / "state" / "ssh" / "id_ed25519" + monkeypatch.setattr(linuxssh, "KEY_PATH", path) + linuxssh._key_cache.clear() + yield path + linuxssh._key_cache.clear() + + +class FakeRemote: + """An SSH server that accepts only the appliance key and answers like the script.""" + + def __init__(self, client_key: asyncssh.SSHKey, power: bool = True, reply: str = ""): + self.client_key = client_key + self.power = power + self.reply = reply # overrides the scripted answer when set + self.commands: list[str] = [] + self.host_key = asyncssh.generate_private_key("ssh-ed25519") + self.server = None + self.port = 0 + + def host_key_line(self) -> str: + return self.host_key.export_public_key("openssh").decode().strip() + + async def __aenter__(self): + remote = self + + class Server(asyncssh.SSHServer): + def begin_auth(self, username): + return True + + def public_key_auth_supported(self): + return True + + def validate_public_key(self, username, key): + return username == "upsshutdown" and ( + key.public_data == remote.client_key.public_data + ) + + async def handle(process: asyncssh.SSHServerProcess): + cmd = process.command or "" + remote.commands.append(cmd) + if remote.reply: + process.stdout.write(remote.reply + "\n") + elif cmd == "pve-ups test": + p = "yes" if remote.power else "no" + process.stdout.write(f"PVE-UPS-REMOTE 1 ok power={p} host=gpu01\n") + elif cmd == "pve-ups shutdown": + process.stdout.write("PVE-UPS-REMOTE 1 accepted\n") + else: + process.stdout.write("PVE-UPS-REMOTE 1 error unknown-command\n") + process.exit(0) + + self.server = await asyncssh.create_server( + Server, "127.0.0.1", 0, server_host_keys=[self.host_key], process_factory=handle + ) + self.port = self.server.sockets[0].getsockname()[1] + return self + + async def __aexit__(self, *exc): + self.server.close() + await self.server.wait_closed() + + def target(self, **kw) -> LinuxHostConfig: + data = dict(name="gpu01", ssh_host="127.0.0.1", ssh_port=self.port) + data.update(kw) + return LinuxHostConfig(**data) + + +# --- key handling --------------------------------------------------------------------- + + +def test_appliance_key_is_generated_private_and_stable(key_path): + line = linuxssh.public_key_line() + assert line.startswith("ssh-ed25519 ") + assert stat.S_IMODE(os.stat(key_path).st_mode) == 0o600 + assert stat.S_IMODE(os.stat(key_path.parent).st_mode) == 0o700 + linuxssh._key_cache.clear() # a restart reads the same key back + assert linuxssh.public_key_line() == line + + +def test_setup_script_carries_the_public_key(key_path): + script = linuxssh.setup_script() + assert linuxssh.SETUP_SCRIPT_KEY_MARKER not in script + assert f"PUBKEY='{linuxssh.public_key_line()}'" in script + + +def test_setup_script_is_valid_posix_sh(key_path, tmp_path): + path = tmp_path / "setup.sh" + path.write_text(linuxssh.setup_script()) + subprocess.run(["sh", "-n", str(path)], check=True) + + +# --- model ---------------------------------------------------------------------------- + + +def test_linux_model_needs_no_api_fields(): + host = LinuxHostConfig(name="gpu01", ssh_host=" 10.0.0.31 ") + assert host.ssh_host == "10.0.0.31" + assert host.address == "ssh://10.0.0.31:22" + assert AppConfig.incomplete_host(host) == "" + assert AppConfig.incomplete_host(LinuxHostConfig(name="x")) == "no SSH address" + assert LinuxHostConfig.secret_fields() == {} + + +def test_config_roundtrip_with_linux_target(tmp_path): + cfg = AppConfig( + hosts=[ + PveHostConfig(name="pve01", api_url="https://10.0.0.10:8006", token_id="a", + token_secret="s"), + LinuxHostConfig(name="gpu01", ssh_host="10.0.0.31", ssh_port=2222, + ssh_host_key="ssh-ed25519 AAAAC3Nz", order=0), + ] + ) + path = tmp_path / "config.yaml" + save_config(cfg, path) + restored = load_config(path) + assert [h.type for h in restored.hosts] == ["pve", "linux"] + gpu = restored.hosts[1] + assert isinstance(gpu, LinuxHostConfig) + assert (gpu.ssh_host, gpu.ssh_port, gpu.ssh_host_key) == ("10.0.0.31", 2222, + "ssh-ed25519 AAAAC3Nz") + + +def test_duplicate_ssh_targets_are_flagged(): + cfg = AppConfig(hosts=[ + LinuxHostConfig(name="a", ssh_host="10.0.0.31"), + LinuxHostConfig(name="b", ssh_host="10.0.0.31"), + LinuxHostConfig(name="c", ssh_host="10.0.0.32"), + ]) + assert cfg.duplicate_api_urls() == ["ssh://10.0.0.31:22"] + + +# --- client against a live SSH server ---------------------------------------------------- + + +async def test_test_connection_reports_power_and_host_key(key_path): + async with FakeRemote(linuxssh.appliance_key()) as remote: + result = await targets.test_connection(remote.target()) + assert result.ok and result.has_power_mgmt + assert "gpu01" in result.message + assert result.host_key == " ".join(remote.host_key_line().split()[:2]) + assert remote.commands == ["pve-ups test"] + + +async def test_missing_sudo_rule_is_not_power(key_path): + async with FakeRemote(linuxssh.appliance_key(), power=False) as remote: + result = await targets.test_connection(remote.target()) + assert result.ok and not result.has_power_mgmt + assert "sudo" in result.message + + +async def test_shutdown_is_accepted(key_path): + async with FakeRemote(linuxssh.appliance_key()) as remote: + ok, msg = await targets.shutdown(remote.target(), timeout=10) + assert ok, msg + assert remote.commands == ["pve-ups shutdown"] + + +async def test_pinned_host_key_is_enforced(key_path): + async with FakeRemote(linuxssh.appliance_key()) as remote: + good = remote.target(ssh_host_key=remote.host_key_line()) + assert (await targets.test_connection(good)).ok + other = asyncssh.generate_private_key("ssh-ed25519").export_public_key().decode() + ok, msg = await targets.shutdown(remote.target(ssh_host_key=other), timeout=10) + assert not ok + assert "does not match the pinned" in msg + assert remote.commands == ["pve-ups test"] # the impostor never got the command + + +async def test_foreign_key_is_refused(key_path): + stranger = asyncssh.generate_private_key("ssh-ed25519") + async with FakeRemote(stranger) as remote: + result = await targets.test_connection(remote.target()) + assert not result.ok + assert "refused the appliance key" in result.message + + +async def test_key_without_forced_command_is_reported(key_path): + async with FakeRemote(linuxssh.appliance_key(), reply="bash: pve-ups: not found") as remote: + result = await targets.test_connection(remote.target()) + assert not result.ok + assert "forced command" in result.message + + +async def test_unreachable_target_fails_within_its_timeout(key_path): + # A listening socket that never speaks SSH: the classic "port open, nothing answers". + server = await asyncio.start_server(lambda r, w: None, "127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + host = LinuxHostConfig(name="mute", ssh_host="127.0.0.1", ssh_port=port) + loop = asyncio.get_running_loop() + started = loop.time() + ok, msg = await targets.shutdown(host, timeout=1.0) + assert not ok + assert loop.time() - started < 1.0 + targets.DEADLINE_GRACE_S + finally: + server.close() + + +async def test_closed_port_fails_cleanly(key_path): + host = LinuxHostConfig(name="off", ssh_host="127.0.0.1", ssh_port=1) + result = await targets.test_connection(host, timeout=3) + assert not result.ok + assert "Cannot connect" in result.message or "No answer" in result.message + + +def test_setup_script_path_is_packaged(): + assert Path(linuxssh.__file__).parent.joinpath("remote", "linux-target-setup.sh").is_file() diff --git a/uv.lock b/uv.lock index c96f73e..7c9ae13 100644 --- a/uv.lock +++ b/uv.lock @@ -80,6 +80,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/42/b9/f8d6fa329ab25128b7e98fd83a3cb34d9db5b059a9847eddb840a0af45dd/argon2_cffi_bindings-25.1.0-cp39-abi3-win_arm64.whl", hash = "sha256:b0fdbcf513833809c882823f98dc2f931cf659d9a1429616ac3adebb49f5db94", size = 27149, upload-time = "2025-07-30T10:01:59.329Z" }, ] +[[package]] +name = "asyncssh" +version = "2.24.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cryptography" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0f/c5/41a0d5477865c48cee65050586092dc3ba3fc1c52e29b47fba08d3a44581/asyncssh-2.24.1.tar.gz", hash = "sha256:efcd36e9b35f79873535b06444a7c9b0a3c61d97081b208c7fdd3fd8a40f1eca", size = 558085, upload-time = "2026-10-04T02:48:24.913Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/07/e5/8bc721f04ff545c5a84c9c23fbf788fbb56960bb57a86c6366bc35be0f66/asyncssh-2.24.1-py3-none-any.whl", hash = "sha256:fc560b4f43be0f0c602d184783e5e3876f5d24d933a25359d86e5a50a5f46fe5", size = 382514, upload-time = "2026-10-04T02:48:23.676Z" }, +] + [[package]] name = "certifi" version = "2026.6.17" @@ -395,6 +408,7 @@ wheels = [ name = "pve-usv" source = { editable = "." } dependencies = [ + { name = "asyncssh" }, { name = "cryptography" }, { name = "fastapi" }, { name = "httpx" }, @@ -416,6 +430,7 @@ dev = [ [package.metadata] requires-dist = [ + { name = "asyncssh", specifier = ">=2.14" }, { name = "cryptography", specifier = ">=43,<50" }, { name = "fastapi", specifier = ">=0.110" }, { name = "httpx", specifier = ">=0.27" },