Home / Architecture
Architecture -How far the architecture goes
-The architecture the project aims for, in three scenarios. The first is the goal of the first release. The second is decided (decision 0020) and not built. The third is a direction with no design yet.
- --
-
- - Keel Linux Standalone - Our goal · not yet released - -
- - Keel Cloud, three nodes - Planned - -
- - Separated tiers - Future - -
Keel Linux Standalone
- Our goal · not yet released -This is what Keel Linux is building first. It runs today from source and test images; there is no release for end users yet.
-One LXC container or VM runs the whole stack: the application, its database and its files. One instance spec describes it, and Keel keeps the machine and the spec in step.
- -keel diff compares and keel spec apply converges.-
-
- Declared and compared.
keel inspectreads the running machine,keel diffshows where it differs from the spec,keel spec applyconverges it and is safe to run again.
- - Network changes that undo themselves. VM only · in keel 0.9.0, not yet released On a VM, a changed network must be confirmed with
keel network confirmfrom a new session within 120 seconds, or the previous configuration comes back. A machine reached only over SSH is the case this is for. In an LXC container the host owns the network; keel compares it and does not change it.
- - Not through Webmin. The network is changed in confconsole or at the command line; Webmin shows it read-only (decision 0019). -
Composition on a mesh
+Keel Linux builds appliances from small components instead of shipping one image per stack. Each component is a Debian package, each machine is one YAML file, and the machines find each other over a WireGuard mesh that is IPv6-first.
+Decided 2026-09-30 spec, inspect, diff, apply working today
Keel Cloud, three nodes
- Planned, decision 0020 -Three nodes, LXC containers or VMs, two or three of them holding the application: one primary takes the writes, the replicas follow it and stand ready to take over. Keel Cloud is optional: three standalone appliances configured by hand do the same, and Keel Cloud only automates the setup; without it, the public name is moved by hand after a failover.
- --
-
- Keel Cloud is optional.
keel-cloudadds DNS with a health check, node discovery and WireGuard key exchange. It is in the spirit of TurnKey's HubDNS, which updates a name but checks no health. An API key for it is offered at installation and never required.
- - The election is its own project,
keel-quorum. One etcd election for the whole appliance: database, files and web tier are promoted together. Every data node is a voter; the third node may be a voter that holds no data, which is not an appliance and runs only etcd and keel-quorum.
- - Private network. The nodes talk over WireGuard. Its configuration is converged under the same 120 second confirmation window as any network change. -
- Status panel. A read-only screen in confconsole shows one line per node: role, replication lag, last file pull, health, and the elected leader. It changes nothing on the other nodes. -
- What it does not promise. Replication and the file pull are asynchronous, so the last seconds before a failure can be lost. DNS takes one to five minutes to follow a failover, and some resolvers ignore low TTLs. In an LXC container there is no watchdog, so fencing relies on the process demoting itself in time; a VM can have one. -
Chosen by the operator when the set is formed
-Made once and stored in etcd, so every node reads the same answer. The same choices exist with or without Keel Cloud.
-| Choice | Options | Default |
|---|---|---|
| Failover: a replica takes over when the primary is lost | automatic, manual | automatic, active only once three voters are present |
| Rejoin: the old primary comes back as a replica | automatic, manual | manual. Automatic first keeps a copy of what the node held |
| Failback: the primary role returns to the original node | automatic, manual, never | manual |
When: after the orchestrated upgrade of milestone M3. Keel Cloud comes last and starts with a decision note of its own, so its design beyond the three services above is not written yet.
-Separated tiers
- Future -The application and the database move to machines of their own. Not designed yet: only the remote database (0013 phase 1) and Galera (0013 phase 4) are planned pieces.
- --
-
- A remote database first. An application pointing at a database on another machine is the first piece (decision 0013, phase 1). It is deferred today. -
- Replication, or Galera. Primary and replicas on both engines; Galera gives MariaDB several writers, with a minimum of three nodes. -
- Sharding is possible, not promised. MariaDB has Spider packaged in Debian 13. PostgreSQL has the pieces but no product: Keel configures the role of the node, and the partition scheme stays with whoever designs the system. -
Composition
+Each service, PostgreSQL, MariaDB, Redis or Nginx, is a first-class component that runs standalone or is consumed by another appliance. The stack that used to be LAMP becomes LEMP (Linux, Nginx, PHP-FPM and a database), and LEMP is a recipe built from components, not an image with an identity of its own.
+What two or more images use is an overlay, kept in common with its configuration and first boot scripts. A named combination with an identity is an appliance.
An appliance, top to bottom
+-
+
- Applicationa manifest and an application inithookPhases 4 and 6 +
- RuntimeKeel PHP, Python, Ruby or Node: Keel Web and the runtimePhase 4 +
- Keel WebCore, nginx, coraza, anubisPhase 3 +
- Keel CoreDebian, wireguard, etcd, crowdsec, the installerPhase 1 +
Data services, beside it
+-
+
- Keel PostgreSQL, Keel MariaDBCore, the database, vip, syncthing +
- Keel RedisCore, redis +
- SearchElasticsearch or OpenSearch, not yet chosen +
- Object storageGarage, as an
s3overlay
+
Embedded in a simple installation; discovered on the mesh in an advanced one. A search index is derived data: it is rebuilt when a node joins and never replicated by file. Decisions 0023, 0033 and 0036.
+Overlays
+An overlay is one component: its packages, its configuration and its first boot scripts. Under decision 0039 each overlay is its own Debian source package, keel-overlay-<name>, released on its own with its own changelog.
| Group | Overlays | Default |
|---|---|---|
| Mesh | wireguard, etcd, vip, syncthing | etcd stopped |
| Protection | crowdsec, coraza, anubis | disabled |
| Web | nginx | |
| Runtimes | php-fpm, python, ruby, nodejs, go | |
| Data | postgresql, mariadb, redis, a search engine | |
| Installer | simple and advanced installation, cloud simple and cloud advanced, discovery, YAML emission and consumption |
Apache is not an overlay: it is retired as the default web server and migrated appliance by appliance. The WordPress test images built today still use it, until Keel PHP exists.
+Appliances
+| Appliance | Composition |
|---|---|
| Keel Core | Debian, wireguard, etcd, crowdsec, the installer |
| Keel Web | Core, nginx, coraza, anubis |
| Keel PostgreSQL, Keel MariaDB | Core, the database, vip, syncthing |
| Keel Redis | Core, redis |
| Keel search | Core, the search engine |
| Keel PHP | Web, php-fpm; replaces LAMP and LAPP |
| Keel Python, Keel Ruby, Keel Node | Web and the runtime |
Applications
+An application appliance declares its runtime, the data services it consumes, the paths of its durable state and its workers. Workers are stateless: they read a queue and write to data services, never to local disk. Planned, Phases 4 and 6
+| Application | Consumes | Replicated state, workers |
|---|---|---|
| WordPress | PHP, MariaDB | wp-content |
| Nextcloud | PHP, MariaDB, Redis, optionally search | data |
| Odoo | Python, PostgreSQL, optionally Redis | filestore and addons; cron and queue workers |
| Mastodon | Ruby, PostgreSQL, Redis, optionally search | media; Sidekiq |
| Ghost | Node, MariaDB | |
| Discourse | Ruby, PostgreSQL, Redis | Sidekiq |
| Gitea or Forgejo | Web, go, PostgreSQL | repositories |
Mastodon is the reference appliance: Ruby, Node.js, PostgreSQL, Redis, Sidekiq and usually object storage and search make it the heaviest of the set, so it is the proof of the composition model. It is built last in its phase, after WordPress, Odoo and Nextcloud (0035).
+The manifest
+Backup, monitoring, upgrades, directory replication and discovery all read the same facts: which paths hold data, which processes to watch, what a node offers. So the manifest format came first. Version 1 is decided, and its reader is the first step of the implementation (tracker#39). Decided, 0041
+-
+
- Two kinds, one schema. An overlay manifest ships in each overlay's package and says what it runs. An appliance manifest ships in each appliance's package and says what it is built on and which overlays it adds. +
- The manifest owns facts, the spec owns choices. A manifest never names a host, an address, a secret file or a count. +
- A state per installation mode. Every overlay is
enabled,disabledoraskin each of the three modes. The image is the same; the mode decides what runs.
+ - Inheritance is additive. Keel Web in a simple installation is one thing for every appliance built on it. +
- Derived, never declared. Monit's checks, the firewall, the backup set, the Syncthing folders and the registry entry are rendered from the manifests. +
- Versioned by an integer, with unknown keys refused at every level. +
# /usr/share/keel/appliances/web.yaml, from keel-web +manifest_version: 1 +kind: appliance +name: web +title: Keel Web +summary: Nginx, Coraza and Anubis, the base of every web appliance +base: core +overlays: + nginx: {simple: enabled, cloud_simple: enabled, cloud_advanced: enabled} + coraza: {simple: disabled, cloud_simple: enabled, cloud_advanced: enabled} + anubis: {simple: disabled, cloud_simple: enabled, cloud_advanced: enabled}+
From the version 1 format. Keel Web has no process of its own; its manifest is three lines of states over Core's.
+The spec: one file is the truth
+The instance spec, /etc/keel/instance.yaml, declares one machine: its names, IPv6 network, certificate policy, users, secrets by reference, database role and monitoring. It is rendered into the variables TurnKey's first boot hooks already read, so no hook has to know that it exists, and the keel command and the console are both clients of the same library. Working today
| Command | What it does |
|---|---|
keel inspect | Write a spec from the running machine or an offline root, every field with its source or the reason it was not inferred. Secrets are never read. |
keel diff | Report drift between the spec and the machine, field by field, through the same collector. |
keel spec apply | Converge what the spec declares, only where it differs. Safe to run again. |
keel spec validate | Report every error in the spec, not just the first. |
keel network confirm | Keep a network change; without it, the change reverts by itself. |
version: 1 +instance: + hostname: blog + fqdn: blog.example.org +network: + interfaces: + eth0: + ipv6: + method: static + address: 2001:db8:1::10/64 + gateway: fe80::1 + nameservers: + - 2001:db8:1::53 +secrets: + root_password: + file: /etc/keel/secrets/root_password+
Decided next (0027, 0041): every installation, simple or advanced, by hand or automated, emits its complete YAML with every default written out, and re-emits it after every upgrade. The spec gains the appliance, the installation mode and the state of every overlay.
+IPv6-first
+Every appliance is reachable on a routable address of its own, such as 2001:db8:4b1::10, with an ACME certificate and its own host keys. No port mapping. IPv4 is optional and never assumed, and the project's examples and defaults use IPv6.
-
+
- Between nodes, native IPv6 goes direct. An IPv4-only node enters the mesh through a rendezvous point that translates (NAT64 with Tayga, in Debian 13). +
- On the mesh,
keel network wireguard suggest-addressprints a random unique local IPv6 address with its /64 for the first node of an overlay.
+ - In the spec, nameservers are listed IPv6 first, and a database listens on literal addresses such as
::1, because on Debianlocalhostis an IPv4-only name.
+
Signed, content-addressed layers
+An appliance is assembled from read-only layers, one per appliance in the chain. Each layer is a deterministic tarball with a sha256, listed in a plain-text layer manifest that is signed by the project key and tied to a git commit. The same commit and manifest give the same bytes, and an update downloads only the layer that changed.
+-
+
keel pullfetches the layers the cache lacks, from a directory or an https URL, checking every digest.
+ keel assembleextracts a cached chain into a root filesystem and packs it as a Proxmox template with its sha512.
+ keel verifychecks installed layers against their manifests.
+ - Layers are published on two channels,
stableandtesting; a third channel would need a decision of its own.
+
Build measurements, on the build host with unmodified TurnKey 19.0 recipes: core layer 326 MB, LAMP stack delta 80 MB, WordPress app delta 133 MB, WordPress rebuilt on cached layers in 69 s instead of 451 s. On the home page.
+Upgrades come from the Keel repository
+Decided, 0039
+-
+
- Everything Keel installs is a
.debin the Keel repository, overlays included, andapt upgradeis the only update mechanism.
+ - Application packages carry a post-install migration hook, such as Odoo's module update or Mastodon's
db:migrate.
+ - Upgrades are ordered across the mesh: a database primary hands over to its standby, upgrades, and takes the role back after catching up. +
- Two tracks,
stableandtesting, declared in the YAML. A simple installation defaults to stable, with unattended upgrades for security only.
+ - The archive is built from a dated, pinned pool, so every image rebuilds from the project's own archive. +