diff --git a/AGENTS.md b/AGENTS.md index 1c2bd8a..c185af8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,9 +8,9 @@ How to work on MLX Runner (a native macOS app by 21no.de). Foundation, Metal, Network, CryptoKit, Security). No SwiftPM packages, no vendored code, no Python — scripts are Swift (`swift scripts/.swift`) or POSIX shell. - **Performance changes need numbers.** Measure with `mlxr bench ` (`MLXR_BENCH_DECODE=1` for - decode only) before and after; keep a change only if it wins. Nothing else may use the GPU while + decode only; for runs use `mlxr run "" --temp 0 --repeat-penalty 1 --seed 42 --max 256`) before and after; keep a change only if it wins. Nothing else may use the GPU while benchmarking. -- **Correctness gates:** `swift test` passes; greedy output stays token-identical; prefill matches +- **Correctness gates:** `swift test` passes; greedy output (`--temp 0 --repeat-penalty 1 --seed 42 --max 256`) stays token-identical (`--repeat-penalty 1` is part of it: 1.x forced 1.0, 2.0 follows the model default); prefill matches incremental decode for every architecture (`prefillMatchesIncrementalDecode`). Model tests run when the model is installed (`mlxr download `), otherwise they skip. The 14B cases need `MLXR_TEST_LARGE=1` (with the Gemma suite they do not fit 16 GB). diff --git a/CHANGELOG.md b/CHANGELOG.md index a5d8685..dde9c20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## 2.0.0 + +The command line now follows Ollama: the same verbs, flags, help layout and error messages, in one process with no background server. The app, the engine and the server are unchanged. Breaking changes for scripts are listed first. + +**Changes that can break a script** + +- **`mlxr run` defaults follow Ollama for every way of giving a prompt** (positional words, `-p`, piped stdin, chat). Sampling comes from the model's `generation_config.json` (validated), else the app's defaults; generation runs until the model stops or the context fills (`--max` / `-n` caps it); no statistics unless `--verbose`. Benchmark, greedy and parity instructions now pass `--temp 0 --repeat-penalty 1 --seed 42 --max 256` explicitly. `--repeat-penalty 1` is part of the greedy command because 1.x forced a repetition penalty of 1.0 while 2.0 follows the model's or the app's default (1.05), which changes greedy text. +- **stdout carries only the payload** (model text, `list`, `show` and `search` rows, the saved path of a `pull`, completion scripts, `bench` output unchanged). Banners, warnings, progress and statistics go to stderr, with `\r` progress only when stderr is a terminal. +- **The default haiku prompt is gone.** `mlxr run ` with no prompt opens the chat on a terminal; with piped input it answers that input. +- **Errors print `Error: ` to stderr and exit 1** instead of crashing (`mlxr download nosuch/repo` used to die with a fatal error, exit 133). A Hugging Face 401 or 404 reads "not found, or private/gated (set HF_TOKEN)". +- `download` is now `pull` (`download` remains an alias); `ls` is an alias of `list`. + +**New** + +- **Ollama commands:** `pull`, `show`, `list`, `rm`, `cp`, `ps`, `stop`, `search`, `run`, `serve`, `completion`; `-h`, `--help` and `mlxr help [command]` print Ollama's layout; bare `mlxr` prints help; `-v` / `--version` print `mlxr version is 2.0.0` without touching Metal or the model library. `create`, `push`, `signin`, `signout` and `launch` are accepted and say they are not supported. `bench` and `tokenize` move under "Developer Commands". +- **Model names:** a Hugging Face repo, an installed short name, a saved preset, or an Ollama-style alias (`name[:tag]`) from the built-in catalog. A missing model is pulled automatically, with Ollama-style progress, when the name resolves to a repo; an unsupported architecture is refused before downloading. +- **`run` flags:** `--verbose`, `--think[=true|false|high|medium|low]`, `--hidethinking`, `--nowordwrap`, `--keepalive` and `--insecure` (accepted, no effect), `--system`, `--temp` (alias `--temperature`), `--top-p`, `--top-k`, `--min-p`, `--repeat-penalty`, `--repeat-last-n`, `--seed`, `-n` / `--max-tokens` (`--max` still works), `--ctx`, `--raw`, `--draft`, `--sparse`, `--spec`, `--ids`, `--no-think`. Options are validated strictly with ranges. Presence and frequency penalty, `--format` and the other options the engine doesn't implement are refused with a clear error. +- **Thinking display as in Ollama:** `Thinking...`, the reasoning, `...done thinking.` and a blank line (dim on a terminal); `--hidethinking` drops it. Word wrap on a terminal, off with `--nowordwrap` or when stdout is not a terminal. +- **Interactive chat** with Ollama's slash commands, `>>> ` and `... ` prompts, multiline input, and line editing and history through the system libedit (no package). A cancelled reply keeps your turn; changing `num_ctx` reloads the model; the history sent to the model is capped. +- **`mlxr upgrade`** (alias `update`) replaces the CLI in place with the latest release, after checking the signed `SHA256SUMS`, the archive digest and the binary's code signature; `--check` only reports. Use `sudo mlxr upgrade` when it is installed in `/usr/local/bin`. 1.x has no such command: reinstall once to get it. +- **Shell completions:** `mlxr completion zsh` and `bash`, generated from the command tables. + +**Deliberate differences from Ollama** + +- Thinking follows each model's own template default (`--think` and `--no-think` override it). +- Ctrl-C during a one-shot reply stops generation and exits 130 (Ollama exits 0). In chat it cancels the current reply and returns to the prompt; at the prompt it prints `Use Ctrl + d or /bye to exit.`. During a download it stops and keeps the partial files, so the next run resumes. +- There is no server to unload from: `ps` lists models loaded by this process only and `keepalive` has no effect. + ## 1.5.0 Qwen 3.5 support: the hybrid Gated DeltaNet architecture. Decode is 6–7% ahead of mlx-lm on the 0.8B model, 2% behind on the 2B and 5% behind on the 4B; prefill is ahead. Thinking now follows each model's own default, and the app keeps its settings across relaunches. diff --git a/Package.swift b/Package.swift index afb851d..b5b40c5 100644 --- a/Package.swift +++ b/Package.swift @@ -17,8 +17,10 @@ let package = Package( swiftSettings: [.unsafeFlags(["-Ounchecked"], .when(configuration: .release))] ), .executableTarget(name: "MLXRunner", dependencies: ["MLXEngine"]), - .executableTarget(name: "mlxr", dependencies: ["MLXEngine"]), + .target(name: "MLXRCLI", dependencies: ["MLXEngine"]), + .executableTarget(name: "mlxr", dependencies: ["MLXRCLI"]), .testTarget(name: "MLXEngineTests", dependencies: ["MLXEngine"]), + .testTarget(name: "MLXRCLITests", dependencies: ["MLXRCLI", "MLXEngine"]), ], swiftLanguageModes: [.v5] ) diff --git a/README.md b/README.md index 5ddef70..c3a2b76 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ sudo mkdir -p /usr/local/bin sudo install -m 755 mlxr--macos-arm64/mlxr /usr/local/bin/mlxr ``` -A browser download is quarantined by macOS and the binary won't start until you run `xattr -d com.apple.quarantine /usr/local/bin/mlxr`. The CLI doesn't update itself: repeat the steps for a new version. +A browser download is quarantined by macOS and the binary won't start until you run `xattr -d com.apple.quarantine /usr/local/bin/mlxr`. From 2.0 on the CLI updates itself: run `mlxr upgrade` (`sudo mlxr upgrade` when it is installed in `/usr/local/bin`). 1.x has no upgrade command, so repeat these steps once to move to 2.0. ### Verifying a download @@ -178,38 +178,41 @@ Models are stored in `~/Library/Application Support/MLX Runner/models// "Explain RoPE" --temp 0 --repeat-penalty 1 --seed 42 --max 256`. Use these +flags for every benchmark or correctness comparison. `--repeat-penalty 1` matters: 1.x forced a repetition penalty of 1.0, while 2.0 uses the model's or the app's default (1.05) unless you pass it. + +Developer commands: + +```bash mlxr bench mlx-community/Qwen3-4B-4bit # prefill/decode throughput, verification cost at short and 1,900-token contexts, kernel profile (MLXR_BENCH_NOPROFILE=1 skips it) mlxr tokenize "some text" MLXR_BENCH_CTX=8192,30000 MLXR_BENCH_TEXT=book.txt mlxr bench # adds dense vs sparse decode after long prompts, with agreement on the text -mlxr run --sparse off # sparse decode attention is on from 5,128 tokens of context; off, on or a key budget (N) +mlxr run "Explain RoPE" --temp 0 --repeat-penalty 1 --seed 42 --max 256 --sparse off # sparse decode attention is on from 5,128 tokens of context; off, on or a key budget (N) ``` -## Architecture - -``` -Sources/ - MLXEngine/ Inference engine (library) - Kernels.swift Metal shader source, compiled at runtime - GPU.swift Device and pipelines; safetensors loader (one aligned copy per shard) - Transformer.swift Model graph: one command buffer per step, fused kernels - SparseAttention.swift Sparse decode attention for long contexts (on by default; docs/sparse-decode-attention.md) - Tokenizer.swift BPE from tokenizer.json (byte-level and SentencePiece) - ChatTemplate.swift ChatML / Llama 3 / Mistral / Phi-3 prompt formats - Engine.swift Generation loop, pipelined decode, KV-prefix reuse, cancellation, context fitting - Hub.swift Hugging Face API client and downloader - ModelStore.swift Local library and HF-cache scanning - OpenAIServer.swift OpenAI-compatible HTTP server (Network.framework) - MLXRunner/ SwiftUI app - mlxr/ CLI -Tests/MLXEngineTests/ Swift Testing suite -``` +Differences from Ollama: thinking follows each model's own template default (`--think` and `--no-think` +override it); Ctrl-C during a reply exits with status 130; `create`, `push`, `signin`, `signout` and +`launch` are accepted and answer that MLX Runner doesn't support them. `mlxr run` without a prompt no +longer writes a haiku: it opens the chat. How the engine works: diff --git a/Sources/MLXEngine/Catalog.swift b/Sources/MLXEngine/Catalog.swift new file mode 100644 index 0000000..9eb7053 --- /dev/null +++ b/Sources/MLXEngine/Catalog.swift @@ -0,0 +1,46 @@ +import Foundation + +/// One curated model with the Ollama-style names it answers to (`qwen3:8b`, `qwen3`, `qwen3:latest`). +public struct CatalogModel: Identifiable, Hashable, Sendable { + /// The Hugging Face repo id. + public let id: String + public let title: String + public let blurb: String + public let gigabytes: Double + /// Lowercase alias names, including the bare family name and `:latest` for the family default. + public let aliases: [String] + + public init(id: String, title: String, blurb: String, gigabytes: Double, aliases: [String] = []) { + self.id = id; self.title = title; self.blurb = blurb; self.gigabytes = gigabytes; self.aliases = aliases + } +} + +/// Curated models, shared by the app's recommendations and the CLI's name resolver. +public struct Catalog: Sendable { + public var models: [CatalogModel] + + public init(models: [CatalogModel]) { self.models = models } + + /// The built-in catalog. + public static let standard = Catalog(models: [ + .init(id: "mlx-community/Qwen3-0.6B-4bit", title: "Qwen3 0.6B", blurb: "Tiny and blazing fast, with thinking mode", gigabytes: 0.35, aliases: ["qwen3:0.6b"]), + .init(id: "mlx-community/Qwen3.5-0.8B-4bit", title: "Qwen3.5 0.8B", blurb: "Newest tiny Qwen: hybrid linear attention, quick direct answers", gigabytes: 0.65, aliases: ["qwen3.5:0.8b", "qwen3.5", "qwen3.5:latest"]), + .init(id: "mlx-community/Llama-3.2-3B-Instruct-4bit", title: "Llama 3.2 3B", blurb: "Meta's compact all-rounder", gigabytes: 1.8, aliases: ["llama3.2:3b", "llama3.2", "llama3.2:latest"]), + .init(id: "mlx-community/Qwen3-4B-4bit", title: "Qwen3 4B", blurb: "Strong reasoning in a small footprint", gigabytes: 2.3, aliases: ["qwen3:4b"]), + .init(id: "mlx-community/Phi-3.5-mini-instruct-4bit", title: "Phi-3.5 mini", blurb: "Microsoft's efficient 3.8B model", gigabytes: 2.2, aliases: ["phi3.5", "phi3.5:latest"]), + .init(id: "mlx-community/Mistral-7B-Instruct-v0.3-4bit", title: "Mistral 7B v0.3", blurb: "Classic, fluent 7B instruct model", gigabytes: 4.1, aliases: ["mistral:7b", "mistral", "mistral:latest"]), + .init(id: "mlx-community/Qwen2.5-Coder-7B-Instruct-4bit", title: "Qwen2.5 Coder 7B", blurb: "Code generation and review", gigabytes: 4.3, aliases: ["qwen2.5-coder:7b", "qwen2.5-coder", "qwen2.5-coder:latest"]), + .init(id: "mlx-community/Qwen3-8B-4bit", title: "Qwen3 8B", blurb: "Best quality under 5 GB", gigabytes: 4.6, aliases: ["qwen3:8b", "qwen3", "qwen3:latest"]), + .init(id: "mlx-community/Meta-Llama-3.1-8B-Instruct-4bit", title: "Llama 3.1 8B", blurb: "Popular 8B with 128K context", gigabytes: 4.5, aliases: ["llama3.1:8b", "llama3.1", "llama3.1:latest"]), + .init(id: "mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit", title: "DeepSeek R1 Distill 7B", blurb: "Step-by-step reasoning", gigabytes: 4.3, aliases: ["deepseek-r1:7b", "deepseek-r1", "deepseek-r1:latest"]), + ]) + + /// The repo id an alias names (case-insensitive), or nil. + public func repo(forAlias alias: String) -> String? { + let key = alias.lowercased() + return models.first { $0.aliases.contains(key) }?.id + } + + /// Every alias, for suggestions and completions. + public var aliases: [String] { models.flatMap(\.aliases) } +} diff --git a/Sources/MLXEngine/Engine.swift b/Sources/MLXEngine/Engine.swift index 1ea8243..1e13f96 100644 --- a/Sources/MLXEngine/Engine.swift +++ b/Sources/MLXEngine/Engine.swift @@ -67,6 +67,19 @@ public struct LoadedModelInfo: Sendable, Hashable { public var sparseNote: String? = nil /// The draft model used for speculative decoding (nil = none). public var draftName: String? = nil + + public init(name: String, directory: URL, modelType: String, quantization: String, parameterBytes: Int, + contextLength: Int, layers: Int, hiddenSize: Int, chatFormat: String, supportsThinking: Bool, + loadSeconds: Double, thinksByDefault: Bool = false, supportsTools: Bool = false, + sparseLayers: Int = 0, sparseTopK: Int = 0, sparseMinContext: Int = 0, + sparseNote: String? = nil, draftName: String? = nil) { + self.name = name; self.directory = directory; self.modelType = modelType; self.quantization = quantization + self.parameterBytes = parameterBytes; self.contextLength = contextLength; self.layers = layers + self.hiddenSize = hiddenSize; self.chatFormat = chatFormat; self.supportsThinking = supportsThinking + self.loadSeconds = loadSeconds; self.thinksByDefault = thinksByDefault; self.supportsTools = supportsTools + self.sparseLayers = sparseLayers; self.sparseTopK = sparseTopK; self.sparseMinContext = sparseMinContext + self.sparseNote = sparseNote; self.draftName = draftName + } } /// Minimal xorshift RNG so sampling is seedable. diff --git a/Sources/MLXEngine/Hub.swift b/Sources/MLXEngine/Hub.swift index a5f97f8..4bf5fa1 100644 --- a/Sources/MLXEngine/Hub.swift +++ b/Sources/MLXEngine/Hub.swift @@ -82,6 +82,12 @@ public struct HubModelDetails: Sendable { public let tags: [String] public let license: String? + public init(id: String, files: [HubFile], modelType: String?, bits: Int?, readmeURL: URL, downloads: Int?, likes: Int?, + tags: [String], license: String?, isMoE: Bool = false) { + self.id = id; self.files = files; self.modelType = modelType; self.bits = bits; self.readmeURL = readmeURL + self.downloads = downloads; self.likes = likes; self.tags = tags; self.license = license; self.isMoE = isMoE + } + public var isMoE: Bool = false public var isSupported: Bool { SupportedArchitectures.isSupported(modelType) && !isMoE && files.contains { $0.rfilename == "tokenizer.json" } diff --git a/Sources/MLXEngine/ModelConfig.swift b/Sources/MLXEngine/ModelConfig.swift index 01c2200..562b31c 100644 --- a/Sources/MLXEngine/ModelConfig.swift +++ b/Sources/MLXEngine/ModelConfig.swift @@ -488,3 +488,42 @@ enum BFloat16Rounding { return Float(bitPattern: rounded) } } + +/// Sampling defaults from a model's generation_config.json. Every value passes through +/// `SamplingLimits`; a value outside them is dropped and described in `warnings`. +public struct GenerationDefaults: Equatable, Sendable { + public var temperature: Float? + public var topP: Float? + public var topK: Int? + public var minP: Float? + public var repetitionPenalty: Float? + /// One line per dropped value, each naming `generation_config.json` and the key. + public var warnings: [String] = [] + + public init() {} + + public static func parse(_ data: Data) -> GenerationDefaults { + var out = GenerationDefaults() + guard let g = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return out } + func number(_ key: String, _ ok: (Double) -> Bool, _ rule: String) -> Double? { + guard let raw = g[key] else { return nil } + if let n = raw as? NSNumber, CFGetTypeID(n) != CFBooleanGetTypeID(), ok(n.doubleValue) { return n.doubleValue } + out.warnings.append("warning: generation_config.json: ignoring \(key) = \(raw) (\(rule))") + return nil + } + out.temperature = number("temperature", { SamplingLimits.temperature.contains($0) }, "must be between 0 and 2").map { Float($0) } + out.topP = number("top_p", SamplingLimits.isValidTopP, "must be greater than 0 and at most 1").map { Float($0) } + out.topK = number("top_k", { $0 == $0.rounded() && SamplingLimits.topK.contains(Int(exactly: $0) ?? -1) }, + "must be a whole number from 0 to \(SamplingLimits.topK.upperBound)").map { Int($0) } + out.minP = number("min_p", { SamplingLimits.minP.contains($0) }, "must be between 0 and 1").map { Float($0) } + out.repetitionPenalty = number("repetition_penalty", SamplingLimits.isValidRepeatPenalty, + "must be greater than 0 and at most 100").map { Float($0) } + return out + } + /// Reads `/generation_config.json`; absent or unreadable gives empty defaults. + public static func load(directory: URL) -> GenerationDefaults { + guard let data = try? Data(contentsOf: directory.appendingPathComponent("generation_config.json"), options: .mappedIfSafe), + data.count <= 1 << 20 else { return GenerationDefaults() } + return parse(data) + } +} diff --git a/Sources/MLXEngine/ModelStore.swift b/Sources/MLXEngine/ModelStore.swift index 8d67f9a..7c8a2bb 100644 --- a/Sources/MLXEngine/ModelStore.swift +++ b/Sources/MLXEngine/ModelStore.swift @@ -17,6 +17,14 @@ public struct LocalModel: Identifiable, Hashable, Sendable { public var isSupported: Bool { SupportedArchitectures.isSupported(modelType) && !isMoE } /// A mixture-of-experts variant of a supported type (Gemma 4 26B-A4B). public var isMoE: Bool = false + /// When the model was last changed on disk. + public var modified: Date? = nil + + public init(repo: String, directory: URL, sizeBytes: Int, modelType: String?, bits: Int?, source: Source, + isMoE: Bool = false, modified: Date? = nil) { + self.repo = repo; self.directory = directory; self.sizeBytes = sizeBytes; self.modelType = modelType + self.bits = bits; self.source = source; self.isMoE = isMoE; self.modified = modified + } } public enum ModelStore { @@ -103,7 +111,7 @@ public enum ModelStore { } var m = LocalModel(repo: repo, directory: dir, sizeBytes: size, modelType: (cfg["model_type"] as? String)?.lowercased(), - bits: (q?["bits"] as? NSNumber)?.intValue, source: source) + bits: (q?["bits"] as? NSNumber)?.intValue, source: source, modified: mtime(dir)) m.isMoE = SupportedArchitectures.isMoE(cfg) return m } diff --git a/Sources/MLXEngine/OpenAIServer.swift b/Sources/MLXEngine/OpenAIServer.swift index 6c005cc..c8403f4 100644 --- a/Sources/MLXEngine/OpenAIServer.swift +++ b/Sources/MLXEngine/OpenAIServer.swift @@ -73,7 +73,7 @@ public final class OpenAIServer: @unchecked Sendable { public var maxBodyBytes = 4 * 1024 * 1024 public var maxMessages = 1000 /// Server-side ceiling for max_tokens / max_completion_tokens / max_output_tokens. - public var maxOutputTokens = 32_768 + public var maxOutputTokens = SamplingLimits.maxTokens public var headerTimeout: TimeInterval = 10 public var bodyTimeout: TimeInterval = 30 public var writeTimeout: TimeInterval = 30 @@ -102,7 +102,7 @@ public final class OpenAIServer: @unchecked Sendable { var maxConnections = 32 var maxQueued = 8 var maxMessages = 1000 - var maxOutputTokens = 32_768 + var maxOutputTokens = SamplingLimits.maxTokens var limits = HTTPConnection.Limits() } @@ -411,7 +411,7 @@ public final class OpenAIServer: @unchecked Sendable { } if let v = try b.number("top_k") { guard v >= -1 else { throw APIError.invalid("'top_k' must be >= 0 (or -1 to disable)", param: "top_k") } - p.topK = v <= 0 ? 0 : Int(min(v, 100_000)) + p.topK = SamplingLimits.clampTopK(v) } if let v = try b.number("min_p") { guard v >= 0 && v <= 1 else { throw APIError.invalid("'min_p' must be between 0 and 1", param: "min_p") } @@ -420,7 +420,7 @@ public final class OpenAIServer: @unchecked Sendable { let rpKey = b.has("repetition_penalty") ? "repetition_penalty" : "repeat_penalty" if let v = try b.number(rpKey) { guard v > 0 else { throw APIError.invalid("'\(rpKey)' must be greater than 0", param: rpKey) } - p.repetitionPenalty = Float(min(v, 100)) + p.repetitionPenalty = Float(SamplingLimits.clampRepeatPenalty(v)) } for k in ["frequency_penalty", "presence_penalty"] { // Accepted for compatibility and range-checked, but not applied (see type docs). @@ -430,7 +430,7 @@ public final class OpenAIServer: @unchecked Sendable { } if let k = maxKeys.first(where: { b.has($0) }), let v = try b.number(k) { guard v >= 1 else { throw APIError.invalid("'\(k)' must be at least 1", param: k) } - p.maxTokens = Int(min(v, Double(cfg.maxOutputTokens))) + p.maxTokens = SamplingLimits.clampMaxTokens(v, cap: cfg.maxOutputTokens) } else { p.maxTokens = min(defaultMax, cfg.maxOutputTokens) } @@ -1480,7 +1480,7 @@ final class OnceFlag: @unchecked Sendable { /// Separates a leading `…` block from the reply while streaming, holding back /// partial tags across chunks. Output that does not start with `` is passed through. -struct ReasoningSplitter { +public struct ReasoningSplitter { private enum State { case start, thinking, content } static let open = "" static let close = "" @@ -1489,9 +1489,9 @@ struct ReasoningSplitter { private var trimContentLead = false private var trimReasoningLead = true - init(enabled: Bool) { state = enabled ? .start : .content } + public init(enabled: Bool) { state = enabled ? .start : .content } - mutating func feed(_ t: String) -> (reasoning: String, content: String) { + public mutating func feed(_ t: String) -> (reasoning: String, content: String) { var reasoning = "" var content = "" pending += t @@ -1539,7 +1539,7 @@ struct ReasoningSplitter { return (reasoning, content) } - mutating func flush() -> (reasoning: String, content: String) { + public mutating func flush() -> (reasoning: String, content: String) { defer { pending = "" } switch state { case .thinking: diff --git a/Sources/MLXEngine/SamplingLimits.swift b/Sources/MLXEngine/SamplingLimits.swift new file mode 100644 index 0000000..edd38f1 --- /dev/null +++ b/Sources/MLXEngine/SamplingLimits.swift @@ -0,0 +1,26 @@ +import Foundation + +/// The bounds of every sampling and serving number, shared by the HTTP server and the `mlxr` CLI. +/// The server clamps some values to these bounds; the CLI rejects values outside them. +public enum SamplingLimits { + public static let temperature: ClosedRange = 0...2 + /// Top-p is valid in (0, topPMax]. + public static let topPMax: Double = 1 + public static let topK: ClosedRange = 0...100_000 + public static let minP: ClosedRange = 0...1 + /// Repeat penalty is valid in (0, repeatPenaltyMax]. + public static let repeatPenaltyMax: Double = 100 + public static let maxTokens: Int = 32_768 + public static let context: ClosedRange = 256...1_048_576 + public static let port: ClosedRange = 1...65_535 + public static let repeatLastN: ClosedRange = 0...4096 + + public static func isValidTopP(_ v: Double) -> Bool { v > 0 && v <= topPMax } + public static func isValidRepeatPenalty(_ v: Double) -> Bool { v > 0 && v <= repeatPenaltyMax } + + /// What the server does with out-of-range values it accepts: top_k and repeat penalty are + /// clamped to their ceiling, max_tokens to the given per-server cap. + public static func clampTopK(_ v: Double) -> Int { v <= 0 ? 0 : Int(min(v, Double(topK.upperBound))) } + public static func clampRepeatPenalty(_ v: Double) -> Double { min(v, repeatPenaltyMax) } + public static func clampMaxTokens(_ v: Double, cap: Int) -> Int { Int(min(v, Double(cap))) } +} diff --git a/Sources/MLXEngine/Updater.swift b/Sources/MLXEngine/Updater.swift index d014034..2b40ca3 100644 --- a/Sources/MLXEngine/Updater.swift +++ b/Sources/MLXEngine/Updater.swift @@ -50,15 +50,20 @@ public struct UpdateInstall: Equatable, Sendable { case macApp /// `swift run` / an unbundled build: updated with git. case source + /// The standalone `mlxr` command-line binary, replaced in place by `mlxr upgrade`. + case cli } public var kind: Kind /// The `.app` bundle that gets replaced. public var target: URL + /// The target is the standalone `mlxr` binary (installed or a source build), not an app bundle. + public var isCommandLine: Bool - public init(kind: Kind, target: URL) { + public init(kind: Kind, target: URL, isCommandLine: Bool? = nil) { self.kind = kind self.target = target + self.isCommandLine = isCommandLine ?? (kind == .cli) } /// Works out how the running app was installed from its bundle location. @@ -67,10 +72,22 @@ public struct UpdateInstall: Equatable, Sendable { return UpdateInstall(kind: url.pathExtension == "app" ? .macApp : .source, target: url) } - /// The release asset for a version. + /// Works out how a running `mlxr` binary was installed: the real path (symlinks resolved) is the + /// target; a binary inside a `.build` directory is a source build. + public static func detectCLI(executable: URL) -> UpdateInstall { + let url = executable.resolvingSymlinksInPath().standardizedFileURL + return UpdateInstall(kind: url.pathComponents.contains(".build") ? .source : .cli, target: url, isCommandLine: true) + } + + /// The release asset for a version (the app's disk image). public static func assetName(for version: String) -> String { "MLX-Runner-\(version)-macos-arm64.dmg" } + + /// The release asset for a version and install kind. + public static func assetName(for version: String, kind: Kind) -> String { + kind == .cli ? "mlxr-\(version)-macos-arm64.tar.gz" : assetName(for: version) + } } /// The result of a successful `Updater.apply`. @@ -300,14 +317,24 @@ public actor Updater { } private func cannotInstall(_ r: Release) -> String? { - if install.kind == .source { return "Running from source: update with git pull." } - let name = UpdateInstall.assetName(for: r.version) + if install.kind == .source { + return install.isCommandLine ? "mlxr is running from a source build; update it with git pull" + : "Running from source: update with git pull." + } + let name = UpdateInstall.assetName(for: r.version, kind: install.kind) if r.assets[name] == nil { return "This release has no build for this Mac (\(name))." } if r.assets["SHA256SUMS"] == nil || r.assets["SHA256SUMS.sig"] == nil { return "This release isn't signed, so it can't be installed automatically." } if Self.publicKey(fromBase64: publicKeyBase64) == nil { return "This build has no update signing key." } let path = install.target.path + if install.kind == .cli { + // The binary is replaced by a rename in its directory, so the directory must be writable. + if !FileManager.default.isWritableFile(atPath: install.target.deletingLastPathComponent().path) { + return "cannot replace \(path) (permission denied); run: sudo mlxr upgrade" + } + return nil + } if path.contains("/AppTranslocation/") || path.hasPrefix("/Volumes/") { return "Move MLX Runner to the Applications folder to update it." } @@ -322,16 +349,17 @@ public actor Updater { /// Downloads, verifies and installs the latest release, reporting each phase. Returns the /// installed version; the caller restarts the app (`Updater.relaunch`). - public func apply(progress: @escaping @Sendable (UpdatePhase) -> Void = { _ in }) async throws -> AppliedUpdate { + /// - Parameter reuseCheck: use the release found by a recent `check()` instead of asking GitHub again. + public func apply(reuseCheck: Bool = false, progress: @escaping @Sendable (UpdatePhase) -> Void = { _ in }) async throws -> AppliedUpdate { guard !busy else { throw UpdateError("An update is already being installed.") } busy = true defer { busy = false } progress(.checking) - let info = try await check(force: true) - guard info.available, let release = cached?.release else { throw UpdateError("MLX Runner is already up to date.") } + let info = try await check(force: !reuseCheck) + guard info.available, let release = cached?.release else { throw UpdateError(install.isCommandLine ? "mlxr is already up to date." : "MLX Runner is already up to date.") } guard info.canInstall else { throw UpdateError(info.reason ?? "This update can't be installed automatically.") } - let name = UpdateInstall.assetName(for: release.version) + let name = UpdateInstall.assetName(for: release.version, kind: install.kind) guard let assetURL = release.assets[name], let sumsURL = release.assets["SHA256SUMS"], let sigURL = release.assets["SHA256SUMS.sig"] else { throw UpdateError("The release is missing files.") } @@ -460,13 +488,17 @@ public actor Updater { let session = URLSession(configuration: configuration, delegate: delegate, delegateQueue: queue) defer { session.finishTasksAndInvalidate() } let task = session.dataTask(with: request) - try await withTaskCancellationHandler { - try await withCheckedThrowingContinuation { (c: CheckedContinuation) in - delegate.continuation = c - task.resume() + do { + try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { (c: CheckedContinuation) in + delegate.continuation = c + task.resume() + } + } onCancel: { + task.cancel() } - } onCancel: { - task.cancel() + } catch let e as URLError where e.code != .cancelled { + throw UpdateError("Network error: \(e.localizedDescription)") } } } @@ -669,3 +701,86 @@ public struct MacAppInstaller: UpdateInstalling { return text } } + + +// MARK: - CLI installer + +/// Extracts `mlxr--macos-arm64/mlxr` from the release tarball with `/usr/bin/tar`, validates it +/// (regular file, size cap, arm64 Mach-O, valid code signature) and swaps it over `target` with one +/// `rename(2)`. Everything it creates in `stage` is removed again, on success and on failure. +public struct CLIInstaller: UpdateInstalling { + public static let defaultMaxBinaryBytes: Int = 256 << 20 + public var maxBinaryBytes: Int + + public init(maxBinaryBytes: Int = CLIInstaller.defaultMaxBinaryBytes) { + self.maxBinaryBytes = maxBinaryBytes + } + + public func install(package: URL, version: String, target: URL, stage: URL) async throws { + // tar and codesign block: keep them off the cooperative thread pool. + try await withCheckedThrowingContinuation { (c: CheckedContinuation) in + DispatchQueue.global(qos: .userInitiated).async { + c.resume(with: Result { try installSync(package: package, version: version, target: target, stage: stage) }) + } + } + } + + func installSync(package: URL, version: String, target: URL, stage: URL) throws { + let folder = "mlxr-\(version)-macos-arm64" + let member = "\(folder)/mlxr" + try checkListing(package: package, folder: folder, member: member) + + let work = stage.appendingPathComponent("extract", isDirectory: true) + try FileManager.default.createDirectory(at: work, withIntermediateDirectories: false) + defer { try? FileManager.default.removeItem(at: work) } + try MacAppInstaller.run("/usr/bin/tar", ["-xzf", package.path, "-C", work.path, "--", member]) + + let staged = work.appendingPathComponent(member) + try validate(staged) + guard chmod(staged.path, 0o755) == 0 else { + throw UpdateError("Could not make the new mlxr executable: \(String(cString: strerror(errno))).") + } + guard rename(staged.path, target.path) == 0 else { + throw UpdateError("Could not replace \(target.path): \(String(cString: strerror(errno))).") + } + } + + /// Every archive member must live under the release folder (no absolute or `..` paths), and the + /// binary must be there exactly once. Other members (LICENSE) are never extracted. + func checkListing(package: URL, folder: String, member: String) throws { + let listing = try MacAppInstaller.run("/usr/bin/tar", ["-tzf", package.path]) + let names = listing.split(separator: "\n", omittingEmptySubsequences: true).map(String.init) + var found = 0 + for name in names { + let parts = name.split(separator: "/", omittingEmptySubsequences: false) + guard !name.hasPrefix("/"), !parts.contains(".."), name.hasPrefix(folder + "/") else { + throw UpdateError("The release archive has an unexpected entry: \(name.prefix(80)).") + } + if name == member { found += 1 } + } + guard found == 1 else { throw UpdateError("The release archive has no \(member).") } + } + + /// A regular file (never a link) under the size cap, arm64 Mach-O, validly code signed. + func validate(_ file: URL) throws { + var st = stat() + guard lstat(file.path, &st) == 0, (st.st_mode & S_IFMT) == S_IFREG else { + throw UpdateError("The new mlxr is not a regular file.") + } + guard st.st_size <= Int64(maxBinaryBytes) else { throw UpdateError("The new mlxr is too large.") } + guard st.st_size >= 8, let handle = try? FileHandle(forReadingFrom: file) else { + throw UpdateError("The new mlxr is not an arm64 executable.") + } + let head = (try? handle.read(upToCount: 8)) ?? Data() + try? handle.close() + // MH_MAGIC_64 (CF FA ED FE), then CPU_TYPE_ARM64 (0x0100000C), both little-endian. + guard Array(head) == [0xCF, 0xFA, 0xED, 0xFE, 0x0C, 0x00, 0x00, 0x01] else { + throw UpdateError("The new mlxr is not an arm64 executable.") + } + do { + try MacAppInstaller.run("/usr/bin/codesign", ["--verify", "--strict", file.path]) + } catch let e as UpdateError { + throw UpdateError("The new mlxr's code signature is invalid: \(e.message)") + } + } +} diff --git a/Sources/MLXEngine/Version.swift b/Sources/MLXEngine/Version.swift index cb5e086..ff33da4 100644 --- a/Sources/MLXEngine/Version.swift +++ b/Sources/MLXEngine/Version.swift @@ -1,7 +1,7 @@ /// The MLX Runner release version. The release workflow refuses a tag that doesn't match it, /// and scripts/bundle.sh writes it into the app's Info.plist. public enum MLXRunnerVersion { - public static let current = "1.5.0" + public static let current = "2.0.0" public static let repository = "BackendStack21/mlx-runner" public static let homepage = "https://mlx-runner.21no.de" } diff --git a/Sources/MLXRCLI/CLI.swift b/Sources/MLXRCLI/CLI.swift new file mode 100644 index 0000000..1ec8398 --- /dev/null +++ b/Sources/MLXRCLI/CLI.swift @@ -0,0 +1,79 @@ +import Foundation +import MLXEngine + +public enum CLI { + /// Parse a command line against a table without running anything. + public static func parse(_ args: [String], commands: [Command] = CLI.commands) throws -> Invocation { + guard let first = args.first else { throw CLIError.usage("accepts a command") } + guard let command = findCommand(first, in: commands) else { throw unknownCommand(first, commands) } + return try parse(args: Array(args.dropFirst()), command: command) + } + + static func unknownCommand(_ name: String, _ commands: [Command]) -> CLIError { + if name.hasPrefix("-") { return .usage("unknown flag: \(name)") } + var msg = "unknown command \"\(name)\" for \"\(rootName)\"" + let near = suggestions(for: name, in: commands) + if !near.isEmpty { msg += "\n\nDid you mean this?\n" + near.map { "\t\($0)\n" }.joined() } + return .usage(msg) + } + + /// Parse and run; returns the process exit code. Errors print `Error: ` on stderr. + public static func run(_ args: [String], console: Console, commands: [Command] = CLI.commands) async -> Int32 { + do { + if let first = args.first { + switch first { + case "-h", "--help": console.out(rootHelp(commands: commands)); return 0 + case "-v", "--version", "version": console.out("mlxr version is \(MLXRunnerVersion.current)\n"); return 0 + case "help": + guard args.count > 1 else { console.out(rootHelp(commands: commands)); return 0 } + guard let c = findCommand(args[1], in: commands) else { + throw CLIError.usage("unknown help topic \"\(args[1])\"") + } + console.out(commandHelp(c)); return 0 + default: break + } + } else { + console.out(rootHelp(commands: commands)); return 0 + } + let inv = try parse(args, commands: commands) + if inv.helpRequested { console.out(commandHelp(inv.command)); return 0 } + try await inv.command.run(inv, console) + return 0 + } catch CLIError.interrupted { + return 130 + } catch CLIError.reported { + return 1 + } catch { + console.err("Error: \(message(for: error))\n") + return 1 + } + } + + /// The message printed for an error (Hugging Face 401/404 become one hint). + public static func message(for error: Error) -> String { + if let e = error as? CLIError { + switch e { + case .usage(let m), .failure(let m): return m + case .interrupted: return "interrupted" + case .reported: return "failed" + } + } + let ns = error as NSError + if ns.domain == "HuggingFace", ns.code == 401 || ns.code == 404 { + return "not found, or private/gated (set HF_TOKEN)" + } + return error.localizedDescription + } + + /// Entry point of the `mlxr` executable. + public static func main(_ args: [String]) async -> Int32 { + setvbuf(stdout, nil, _IONBF, 0) + Signals.install() + let console = Console( + out: { FileHandle.standardOutput.write(Data($0.utf8)) }, + err: { FileHandle.standardError.write(Data($0.utf8)) }, + stdoutIsTTY: isatty(1) != 0, stderrIsTTY: isatty(2) != 0, stdinIsTTY: isatty(0) != 0, + env: ProcessInfo.processInfo.environment) + return await run(args, console: console) + } +} diff --git a/Sources/MLXRCLI/Chat.swift b/Sources/MLXRCLI/Chat.swift new file mode 100644 index 0000000..c77cacc --- /dev/null +++ b/Sources/MLXRCLI/Chat.swift @@ -0,0 +1,526 @@ +import Foundation +import MLXEngine + +/// What the chat prompt returned. +public enum LineEvent: Equatable, Sendable { + /// A line of input (without the newline). + case line(String) + /// Ctrl-C at the prompt. + case interrupt + /// Ctrl-D on an empty line, or the end of piped input. + case eof +} + +/// Where the chat loop reads lines from: libedit on a terminal, `readLine()` otherwise, a script in tests. +public protocol LineSource: AnyObject { + /// Shows `prompt` (`>>> ` or `... `) and returns the next event. + func read(prompt: String) -> LineEvent +} + +/// The `readLine()` fallback used when stdin or stdout is not a terminal: each line is one event, +/// the end of input is `.eof`. Writes nothing to stdout. +public final class ConsoleLineSource: LineSource { + private let console: Console + public init(console: Console) { self.console = console } + public func read(prompt: String) -> LineEvent { + console.readLine().map { .line($0) } ?? .eof + } +} + +/// The chat input history: one line per entry in a file under Application Support/MLX Runner/. +public struct ChatHistory: Sendable { + /// The history file; nil keeps no history. + public var file: URL? + public init(file: URL?) { self.file = file } + + private static let sizeLimit = 4 << 20 + + /// `Application Support/MLX Runner/history`. + public static var live: ChatHistory { + ChatHistory(file: ModelStore.root.deletingLastPathComponent().appendingPathComponent("history")) + } + + /// Keeps the file bounded: past the cap, rewrites it with the newest lines (about half the cap). + private func trimIfOversized(_ file: URL) { + guard let size = (try? FileManager.default.attributesOfItem(atPath: file.path)[.size] as? NSNumber)?.intValue, + size > Self.sizeLimit else { return } + let kept = tail(file, keep: Self.sizeLimit / 2).joined(separator: "\n") + "\n" + let tmp = file.deletingLastPathComponent().appendingPathComponent(".history-\(UUID().uuidString)") + guard FileManager.default.createFile(atPath: tmp.path, contents: Data(kept.utf8), attributes: [.posixPermissions: 0o600]) else { return } + if (try? FileManager.default.replaceItemAt(file, withItemAt: tmp)) == nil { try? FileManager.default.removeItem(at: tmp) } + } + + /// The newest lines of the file (at most the last `sizeLimit` bytes), oldest first. + private func tail(_ file: URL, keep: Int) -> [String] { + guard let handle = try? FileHandle(forReadingFrom: file) else { return [] } + defer { try? handle.close() } + guard let end = try? handle.seekToEnd() else { return [] } + let start = end > UInt64(keep) ? end - UInt64(keep) : 0 + guard (try? handle.seek(toOffset: start)) != nil, let data = try? handle.readToEnd() else { return [] } + var lines = String(decoding: data, as: UTF8.self).split(separator: "\n").map(String.init) + if start > 0, !lines.isEmpty { lines.removeFirst() } // a cut-off first line + return lines + } + + /// The saved lines, oldest first (the newest ones when the file is larger than the cap). + public func lines() -> [String] { + guard let file else { return [] } + return tail(file, keep: Self.sizeLimit) + } + + /// Adds a line (creates the file with mode 0600). Failures are ignored: history is a convenience. + public func append(_ line: String) { + guard let file else { return } + let fm = FileManager.default + try? fm.createDirectory(at: file.deletingLastPathComponent(), withIntermediateDirectories: true) + if !fm.fileExists(atPath: file.path) { + fm.createFile(atPath: file.path, contents: nil, attributes: [.posixPermissions: 0o600]) + } + guard let handle = try? FileHandle(forWritingTo: file) else { return } + defer { try? handle.close() } + _ = try? handle.seekToEnd() + let flat = line.replacingOccurrences(of: "\n", with: " ").replacingOccurrences(of: "\r", with: " ") + try? handle.write(contentsOf: Data((flat + "\n").utf8)) + try? handle.close() + trimIfOversized(file) + } +} + +/// The state a chat starts with: the loaded model, the sampling parameters and the display flags. +public struct ChatSettings { + /// The model name as typed. + public var model: String + public var repo: String? + public var directory: URL + public var info: LoadedModelInfo + public var preset: Preset? + public var parameters = SamplingParameters() + /// Parameters set by the user (flags, `/set parameter`, a preset), by Ollama key, as typed. + public var userParameters: [String: String] = [:] + public var system: String? + /// nil: the template's default. + public var thinking: Bool? + public var hideThinking = false + public var wordWrap = true + public var verbose = false + public var contextLength = 4096 + public var draftDirectory: URL? + public var sparse: SparseAttention? + + public init(model: String, repo: String?, directory: URL, info: LoadedModelInfo, preset: Preset? = nil) { + self.model = model; self.repo = repo; self.directory = directory; self.info = info; self.preset = preset + } +} + +/// `/set parameter` keys: Ollama's, with what the engine does with each. +enum ChatParameter { + static let noEffect: Set = ["num_gpu", "main_gpu", "use_mmap", "num_thread", "num_batch", + "typical_p", "num_keep", "stop", "draft_num_predict"] + static let unsupported: Set = ["presence_penalty", "frequency_penalty"] + + enum Outcome { case set, ignored } + + /// Validates `value` against the shared limits and applies it; throws the reason. + static func apply(_ key: String, _ value: String, to p: inout SamplingParameters) throws -> Outcome { + func real(_ range: ClosedRange) throws -> Float { + guard let v = Double(value), v.isFinite, range.contains(v) else { + throw CLIError.failure("invalid value '\(value)' for \(key) (\(range.lowerBound) to \(range.upperBound))") + } + return Float(v) + } + func int(_ range: ClosedRange) throws -> Int { + guard let v = Int(value), range.contains(v) else { + throw CLIError.failure("invalid value '\(value)' for \(key) (\(range.lowerBound) to \(range.upperBound))") + } + return v + } + switch key { + case "temperature": p.temperature = try real(SamplingLimits.temperature) + case "top_k": p.topK = try int(SamplingLimits.topK) + case "min_p": p.minP = try real(SamplingLimits.minP) + case "top_p": + guard let v = Double(value), v.isFinite, SamplingLimits.isValidTopP(v) else { + throw CLIError.failure("invalid value '\(value)' for top_p (above 0, up to 1)") + } + p.topP = Float(v) + case "repeat_penalty": + guard let v = Double(value), v.isFinite, SamplingLimits.isValidRepeatPenalty(v) else { + throw CLIError.failure("invalid value '\(value)' for repeat_penalty (above 0, up to 100)") + } + p.repetitionPenalty = Float(v) + case "repeat_last_n": p.repetitionWindow = try int(SamplingLimits.repeatLastN) + case "num_predict": p.maxTokens = try int(1...SamplingLimits.maxTokens) + case "seed": + guard let v = UInt64(value) else { throw CLIError.failure("invalid value '\(value)' for seed") } + p.seed = v + case "num_ctx": _ = try int(SamplingLimits.context) + case _ where noEffect.contains(key): return .ignored + case _ where unsupported.contains(key): throw CLIError.failure("\(key) is not supported by the engine") + default: throw CLIError.failure("unknown parameter '\(key)'") + } + return .set + } +} + +/// The interactive chat loop (`mlxr run MODEL` on a terminal). +public final class ChatSession { + private let console: Console + private let engine: GenerationEngine + private let resolver: ModelResolver + private let presets: PresetStore + private let interrupts: Interrupts + private let lines: LineSource + private let history: ChatHistory + private var settings: ChatSettings + private var messages: [ChatMessage] = [] + private var recordHistory: Bool + private var reachedEnd = false + /// True once the chat ended because the engine could not load a model (already reported). + public private(set) var failed = false + + public init(console: Console, engine: GenerationEngine, resolver: ModelResolver, presets: PresetStore, + interrupts: Interrupts, lines: LineSource, history: ChatHistory, settings: ChatSettings) { + self.console = console; self.engine = engine; self.resolver = resolver; self.presets = presets + self.interrupts = interrupts; self.lines = lines; self.history = history; self.settings = settings + let flag = (console.env["OLLAMA_NOHISTORY"] ?? "").lowercased() + recordHistory = flag.isEmpty || flag == "0" || flag == "false" + } + + private func say(_ s: String) { console.err(s + "\n") } + + /// Reads and handles input until `/bye`, `/exit` or Ctrl-D; returns the exit code. + @discardableResult + public func run() async -> Int32 { + interrupts.enter(.idle) + while true { + switch lines.read(prompt: ">>> ") { + case .eof: return 0 + case .interrupt: say("\nUse Ctrl + d or /bye to exit.") + case .line(let raw): + let line = raw.trimmingCharacters(in: .whitespaces) + if line.isEmpty { continue } + if recordHistory { history.append(raw) } + if line.hasPrefix("\"\"\"") { + guard let text = gather(line) else { if reachedEnd { return 0 }; continue } + await send(text) + } else if line.hasPrefix("/") { + if let code = await command(line) { return code } + } else { + await send(line) + } + } + if reachedEnd { return 0 } + } + } + + // MARK: input + + /// A triple-quoted message: `first` starts with `"""`; lines join with newlines until one ends with + /// `"""`. Ctrl-C discards it (nil); end of input discards it and ends the chat. + private func gather(_ first: String) -> String? { + let q = "\"\"\"" + var rest = String(first.dropFirst(3)) + var parts: [String] = [] + if rest.hasSuffix(q) { return String(rest.dropLast(3)) } + if !rest.isEmpty { parts.append(rest) } + while true { + switch lines.read(prompt: "... ") { + case .interrupt: return nil + case .eof: reachedEnd = true; return nil + case .line(let l): + rest = l + if rest.hasSuffix(q) { + let before = String(rest.dropLast(3)) + if !before.isEmpty || !parts.isEmpty { parts.append(before) } + return parts.joined(separator: "\n") + } + parts.append(rest) + } + } + } + + // MARK: a turn + + private func send(_ text: String) async { + messages.append(ChatMessage(role: .user, content: text)) + let request = (settings.system.map { [ChatMessage(role: .system, content: $0)] } ?? []) + messages + let token = CancelToken() + let userCancel = UserCancel() + interrupts.enter(.generating, onCancel: { userCancel.mark(); token.cancel() }) + let stream = engine.chat(request, parameters: settings.parameters, enableThinking: settings.thinking, token: token) + let renderer = ResponseRenderer(console: console, hideThinking: settings.hideThinking, wordWrap: settings.wordWrap) + var splitter = ReasoningSplitter(enabled: settings.thinking != false) + var answer = "" + var promptStats: GenerationStats? + var final: GenerationStats? + func emit(_ parts: (reasoning: String, content: String)) { + renderer.reasoning(parts.reasoning) + renderer.content(parts.content) + answer += parts.content + } + var failure: Error? + do { + for try await ev in stream { + switch ev { + case .promptProcessed(let s): promptStats = s + case .text(let t): emit(splitter.feed(t)) + case .finished(let s): final = s + } + } + } catch { failure = error } + interrupts.enter(.idle) + let cancelled = userCancel.isSet || final?.stopReason == "cancelled" + if failure == nil && !cancelled { emit(splitter.flush()) } + if cancelled || failure != nil { + if !cancelled { messages.removeLast() } // a cancelled turn keeps the user message + console.out("\n") + if !cancelled, let failure { console.err("Error: \(CLI.message(for: failure))\n") } + return + } + messages.append(ChatMessage(role: .assistant, content: answer)) + console.out("\n\n") + if settings.verbose, let final { + console.err(StatsFormat.block(RunStats(loadSeconds: 0, stats: final))) + StatsFormat.mlxrLines(prompt: promptStats, final: final).forEach { console.err($0 + "\n") } + } + } + + // MARK: slash commands + + private static let help = """ + Available Commands: + /set Set session variables + /show Show model information + /load Load a session or model + /save Save your current session + /clear Clear session context + /bye Exit + /?, /help Help for a command + /? shortcuts Help for keyboard shortcuts + + Use \"\"\" to begin a multi-line message. + + + """ + + private static let shortcuts = """ + Available Keyboard Shortcuts: + Ctrl + a Move to the beginning of the line (Home) + Ctrl + e Move to the end of the line (End) + Alt + b Move back (left) one word + Alt + f Move forward (right) one word + Ctrl + k Delete the sentence after the cursor + Ctrl + u Delete the sentence before the cursor + Ctrl + w Delete the word before the cursor + Ctrl + l Clear the screen + Ctrl + c Stop the model from responding + Ctrl + d Exit mlxr (/bye) + + + """ + + private func unknown(_ text: String) { say("Unknown command '\(text)'. Type /? for help") } + + /// Handles one slash command; returns an exit code when the chat ends. + private func command(_ line: String) async -> Int32? { + let words = line.split(separator: " ", omittingEmptySubsequences: true).map(String.init) + let name = words[0] + let args = Array(words.dropFirst()) + switch name { + case "/?", "/help": + console.err(args.first == "shortcuts" ? Self.shortcuts : Self.help) + case "/bye", "/exit": return 0 + case "/clear": + messages = [] + say("Cleared session context") + case "/set": return await set(line, args) + case "/show": show(args) + case "/load": return await load(args.joined(separator: " ")) + case "/save": save(args.first) + default: unknown(name) + } + return nil + } + + private func set(_ line: String, _ args: [String]) async -> Int32? { + guard let sub = args.first else { + say("Usage:\n /set parameter ...\n /set system \n /set history|nohistory\n /set wordwrap|nowordwrap\n /set format json\n /set noformat\n /set verbose|quiet\n /set think|nothink") + return nil + } + switch sub { + case "parameter": + guard args.count >= 3 else { say("Usage:\n /set parameter "); return nil } + return await setParameter(args[1], args.dropFirst(2).joined(separator: " ")) + case "system": + guard let range = line.range(of: "system") else { return nil } + let rest = line[range.upperBound...].trimmingCharacters(in: .whitespaces) + guard !rest.isEmpty else { say("Usage:\n /set system "); return nil } + if rest.hasPrefix("\"\"\"") { + guard let text = gather(rest) else { return nil } + settings.system = text + } else { + settings.system = rest + } + say("Set system message.") + case "history": recordHistory = true + case "nohistory": recordHistory = false + case "wordwrap": settings.wordWrap = true; say("Set 'wordwrap' mode.") + case "nowordwrap": settings.wordWrap = false; say("Set 'nowordwrap' mode.") + case "verbose": settings.verbose = true; say("Set 'verbose' mode.") + case "quiet": settings.verbose = false; say("Set 'quiet' mode.") + case "format": + if args.count == 2, args[1] == "json" { + say("Set format to 'json' mode.") + say("warning: format has no effect: not supported by the engine") + } else { + say("Invalid or missing format. For 'json' mode use '/set format json'") + } + case "noformat": say("Disabled format.") + case "think": + settings.thinking = true + if args.count > 1, ["high", "medium", "low"].contains(args[1]) { say("Set 'think' mode to '\(args[1])'.") } + else { say("Set 'think' mode.") } + case "nothink": + settings.thinking = false + say("Set 'nothink' mode.") + default: unknown("/set \(sub)") + } + return nil + } + + private func setParameter(_ key: String, _ value: String) async -> Int32? { + var p = settings.parameters + do { + switch try ChatParameter.apply(key, value, to: &p) { + case .set: + if key == "num_ctx", let ctx = Int(value), ctx != settings.contextLength, + let code = await reload(contextLength: ctx) { return code } + settings.parameters = p + settings.userParameters[key] = value + say("Set parameter '\(key)' to '\(value)'") + case .ignored: + say("warning: parameter '\(key)' has no effect: mlxr does not use it") + } + } catch { + say("Couldn't set parameter: \"\(CLI.message(for: error))\"") + } + return nil + } + + /// Reloads the current model with a new context length; the conversation is kept. + private func reload(contextLength: Int) async -> Int32? { + interrupts.enter(.loading) + defer { interrupts.enter(.idle) } + do { + settings.info = try await engine.load(directory: settings.directory, draftDirectory: settings.draftDirectory, + contextLength: contextLength, sparse: settings.sparse) + settings.contextLength = contextLength + return nil + } catch { + say("Error: \(CLI.message(for: error))") + failed = true + return 1 + } + } + private func show(_ args: [String]) { + guard let sub = args.first else { + say("Available Commands:\n /show info Show details for this model\n /show license Show model license\n /show modelfile Show Modelfile for this model\n /show parameters Show parameters for this model\n /show system Show system message\n /show template Show prompt template") + return + } + guard ["info", "license", "modelfile", "parameters", "system", "template"].contains(sub) else { + unknown("/show \(sub)"); return + } + let info: ShowInfo + do { info = try ShowInfo.read(directory: settings.directory, repo: settings.repo, preset: nil) } catch { + say("Error: \(CLI.message(for: error))"); return + } + switch sub { + case "info": console.err(info.tables()) + case "license": console.out(info.license.map(CLI.endingInNewline) ?? "No license was specified for this model.\n") + case "template": console.out(info.template.map(CLI.endingInNewline) ?? "No prompt template was specified for this model.\n") + case "modelfile": console.out(info.modelfile()) + case "system": + let text = settings.system ?? info.system + console.out(text.map(CLI.endingInNewline) ?? "No system message was specified for this model.\n") + default: + func rows(_ kv: [(key: String, value: String)]) -> String { + kv.map { $0.key.padding(toLength: 30, withPad: " ", startingAt: 0) + " " + $0.value + "\n" }.joined() + } + console.out("Model defined parameters:\n" + rows(info.parameters) + "\nUser defined parameters:\n" + + rows(settings.userParameters.sorted { $0.key < $1.key }.map { ($0.key, $0.value) })) + } + } + + private func save(_ name: String?) { + guard let name else { say("Usage:\n /save "); return } + guard let repo = settings.repo else { say("Couldn't save: the model has no repository id"); return } + do { + try presets.save(Preset(name: name, repo: repo, system: settings.system, parameters: settings.userParameters)) + say("Created new model '\(name)'") + } catch { + say(CLI.message(for: error)) + } + } + + private func load(_ name: String) async -> Int32? { + guard !name.isEmpty else { say("Usage:\n /load "); return nil } + guard let r = try? resolver.resolve(name), let dir = r.directory else { + say("Couldn't find model '\(name)'") + return nil + } + say("Loading model '\(name)'") + interrupts.enter(.loading) + do { + let info = try await engine.load(directory: dir, draftDirectory: settings.draftDirectory, + contextLength: settings.contextLength, sparse: settings.sparse) + interrupts.enter(.idle) + var next = settings + next.model = name; next.repo = r.repo; next.directory = dir; next.info = info; next.preset = r.preset + next.parameters = CLI.samplingDefaults(directory: dir, console: console) + next.userParameters = [:] + next.system = r.preset?.system + settings = next + messages = [] + for (k, v) in (r.preset?.parameters ?? [:]).sorted(by: { $0.key < $1.key }) { + var p = settings.parameters + if (try? ChatParameter.apply(k, v, to: &p)) == .set { + settings.parameters = p + settings.userParameters[k] = v + } + } + return nil + } catch { + interrupts.enter(.idle) + say("Error: \(CLI.message(for: error))") + failed = true + return 1 + } + } +} + +extension ChatParameter.Outcome: Equatable {} + +extension CLI { + /// The model's own sampling defaults (validated generation_config.json) over the engine's. + static func samplingDefaults(directory: URL, console: Console) -> SamplingParameters { + let defaults = GenerationDefaults.load(directory: directory) + defaults.warnings.forEach { console.err($0 + "\n") } + var params = SamplingParameters() + if let v = defaults.temperature { params.temperature = v } + if let v = defaults.topP { params.topP = v } + if let v = defaults.topK { params.topK = v } + if let v = defaults.minP { params.minP = v } + if let v = defaults.repetitionPenalty { params.repetitionPenalty = v } + params.maxTokens = SamplingLimits.maxTokens + return params + } + + /// The chat input for this terminal: libedit when stdin and stdout are terminals, else `readLine()`. + static func liveLineSource(console: Console, history: ChatHistory) -> LineSource { + if console.stdinIsTTY && console.stdoutIsTTY { + let noHistory = ["1", "true"].contains((console.env["OLLAMA_NOHISTORY"] ?? "").lowercased()) + return LibeditLineSource(history: noHistory ? [] : history.lines(), console: console) + } + return ConsoleLineSource(console: console) + } +} diff --git a/Sources/MLXRCLI/Commands.swift b/Sources/MLXRCLI/Commands.swift new file mode 100644 index 0000000..a1bbbb9 --- /dev/null +++ b/Sources/MLXRCLI/Commands.swift @@ -0,0 +1,129 @@ +import Foundation +import MLXEngine + +extension CLI { + static let sparseHelp = "sparse decode attention: on, off or a number of long-range keys" + static let draftHelp = "draft model for speculative decoding: a model or auto" + + /// The command table of `mlxr`. + public static var commands: [Command] { commands() } + + /// The command table with its outside world injected. + public static func commands(services: CLIServices = .live, library: ModelLibrary = .live, presets: PresetStore = .live, + catalog: Catalog = .standard, now: @escaping @Sendable () -> Date = { Date() }, + hub: HubServices = .live, engine: GenerationEngine = LiveGenerationEngine(), + interrupts: Interrupts = .shared, lineSource: LineSource? = nil, + chatHistory: ChatHistory = .live, + updater: @escaping @Sendable () -> Updater = { CLI.liveUpdater() }) -> [Command] { + let runDeps = RunDependencies(resolver: ModelResolver(library: library, presets: presets, catalog: catalog), hub: hub, + engine: engine, interrupts: interrupts, lineSource: lineSource, history: chatHistory) + let ctxRange = SamplingLimits.context + let table: [Command] = [ + Command(name: "run", summary: "Run a model", args: [Arg("MODEL", kind: .model), Arg("PROMPT", optional: true, variadic: true)], + options: [ + Opt(long: "prompt", short: "p", value: .text, help: "prompt"), + Opt(long: "max-tokens", aliases: ["max"], short: "n", value: .int(1...SamplingLimits.maxTokens), help: "maximum tokens to generate (default: until the model stops)"), + Opt(long: "temp", aliases: ["temperature"], value: .real(SamplingLimits.temperature), help: "sampling temperature (default: the model's)"), + Opt(long: "top-p", value: .realAboveZero(max: SamplingLimits.topPMax), help: "nucleus sampling probability"), + Opt(long: "top-k", value: .int(SamplingLimits.topK), help: "sample from the k likeliest tokens (0 turns it off)"), + Opt(long: "min-p", value: .real(SamplingLimits.minP), help: "minimum probability relative to the likeliest token"), + Opt(long: "repeat-penalty", value: .realAboveZero(max: SamplingLimits.repeatPenaltyMax), help: "repetition penalty (1 = off)"), + Opt(long: "repeat-last-n", value: .int(SamplingLimits.repeatLastN), help: "tokens the repetition penalty looks back over"), + Opt(long: "seed", value: .uint64, help: "random seed"), + Opt(long: "system", value: .text, help: "system prompt"), + Opt(long: "ctx", value: .int(ctxRange), defaultText: "4096", help: "context length"), + Opt(long: "raw", help: "send the prompt without the chat template"), + Opt(long: "think", value: .choice(["true", "false", "high", "medium", "low"]), implicit: "true", help: "think before answering"), + Opt(long: "no-think", help: "answer without thinking"), + Opt(long: "ids", help: "print the prompt token ids"), + Opt(long: "draft", value: .model, help: draftHelp), + Opt(long: "sparse", value: .text, help: sparseHelp), + Opt(long: "spec", help: "speculative decoding with n-gram drafts"), + Opt(long: "verbose", help: "print timing statistics to stderr"), + Opt(long: "hidethinking", help: "do not show the model's thinking"), + Opt(long: "nowordwrap", help: "do not wrap words to the next line automatically"), + Opt(long: "keepalive", value: .text, help: "accepted for Ollama compatibility; has no effect"), + Opt(long: "insecure", help: "accepted for Ollama compatibility; has no effect"), + Opt(long: "format", value: .text, help: "not supported by the engine"), + Opt(long: "truncate", help: "not supported (embedding models)"), + Opt(long: "dimensions", value: .int(0...Int.max), help: "not supported (embedding models)"), + Opt(long: "voice", value: .text, help: "not supported (speech models)"), + Opt(long: "ask", value: .text, repeatable: true, help: "not supported"), + Opt(long: "choice", value: .text, repeatable: true, help: "not supported"), + Opt(long: "level", value: .text, repeatable: true, help: "not supported"), + Opt(long: "presence-penalty", value: .text, help: "not supported by the engine"), + Opt(long: "frequency-penalty", value: .text, help: "not supported by the engine"), + ], + examples: [" mlxr run mlx-community/Qwen3-4B-4bit -p \"Hello\" --temp 0"], + environment: [("MLXR_SPARSE_ATTN", "sparse decode attention default (0 turns it off)")], + run: { inv, console in try await executeRun(inv, console, runDeps) }), + Command(name: "serve", aliases: ["start"], summary: "Start the OpenAI-compatible server", args: [Arg("MODEL", kind: .model, optional: true)], + options: [ + Opt(long: "port", value: .int(SamplingLimits.port), defaultText: "1234", help: "port to listen on"), + Opt(long: "ctx", value: .int(ctxRange), defaultText: "8192", help: "context length"), + Opt(long: "lan", help: "listen on the local network (needs an API key)"), + Opt(long: "api-key", value: .text, help: "API key clients must send"), + Opt(long: "draft", value: .model, help: draftHelp), + Opt(long: "sparse", value: .text, help: sparseHelp), + ], + environment: [("MLXR_API_KEY", "API key, when --api-key is not given")], + run: { inv, _ in + guard let model = inv.positionals.first else { + throw CLIError.failure("serve needs a model for now: mlxr serve MODEL (serving without a model comes with the background server in a later release)") + } + try await legacyServe(model: model, port: UInt16(inv.int("port") ?? 1234), + ctx: inv.int("ctx") ?? 8192, lan: inv.bool("lan"), apiKey: inv.string("api-key"), + draft: inv.string("draft"), sparse: inv.string("sparse")) + }), + pullCommand(library: library, presets: presets, catalog: catalog, hub: hub), + showCommand(library: library, presets: presets, catalog: catalog), + Command(name: "search", summary: "Search Hugging Face for MLX models", args: [Arg("QUERY", variadic: true)], + options: [ + Opt(long: "limit", value: .int(1...100), defaultText: "25", help: "maximum number of results"), + Opt(long: "json", help: "print the results as JSON"), + ], + exactArgWording: true, + run: { inv, console in + let models = try await services.search(inv.positionals.joined(separator: " "), inv.int("limit") ?? 25) + console.out(renderSearch(models, json: inv.bool("json"))) + }), + Command(name: "tokenize", summary: "Tokenize text with a model's tokenizer", args: [Arg("MODEL", kind: .model), Arg("TEXT", variadic: true)], + options: [Opt(long: "ids", help: "print only the token ids")], + group: "Developer", run: { inv, console in + try await services.tokenize(TokenizeOptions(model: inv.positionals[0], text: inv.positionals.dropFirst().joined(separator: " "), + idsOnly: inv.bool("ids")), console) + }), + Command(name: "bench", summary: "Benchmark a model", args: [Arg("MODEL", kind: .model)], + options: [ + Opt(long: "decode", help: "decode benchmark only"), + Opt(long: "ctx", value: .text, help: "long-context contexts, comma separated"), + ], + environment: [("MLXR_BENCH_DECODE", "decode benchmark only"), ("MLXR_BENCH_CTX", "long-context contexts, comma separated")], + group: "Developer", run: { inv, console in + try await services.bench(BenchOptions(model: inv.positionals[0], decodeOnly: inv.bool("decode"), contexts: inv.string("ctx")), console) + }), + ] + modelCommands(library: library, presets: presets, catalog: catalog, now: now) + [upgradeCommand(updater: updater)] + return table + [completionCommand(table: table)] + } +} + +extension CLI { + /// Search results: one row per model, or a JSON array with `--json`. + static func renderSearch(_ models: [HubModel], json: Bool) -> String { + if json { + let rows: [[String: Any]] = models.map { m in + var r: [String: Any] = ["id": m.id] + if let d = m.downloads { r["downloads"] = d } + if let l = m.likes { r["likes"] = l } + if let t = m.modelTypeTag { r["model_type"] = t } + if let u = m.lastModified { r["last_modified"] = u } + return r + } + let data = (try? JSONSerialization.data(withJSONObject: rows, options: [.prettyPrinted, .sortedKeys])) ?? Data("[]".utf8) + return String(decoding: data, as: UTF8.self) + "\n" + } + return models.map { + String(format: "%-60@ %10d ↓ %@", $0.id as NSString, $0.downloads ?? 0, ($0.modelTypeTag ?? "") as NSString) + "\n" + }.joined() + } +} diff --git a/Sources/MLXRCLI/Completions.swift b/Sources/MLXRCLI/Completions.swift new file mode 100644 index 0000000..b15e4b6 --- /dev/null +++ b/Sources/MLXRCLI/Completions.swift @@ -0,0 +1,128 @@ +import Foundation + +/// Shell completion scripts generated from the command tables. +extension CLI { + static let completionShells = ["zsh", "bash"] + + /// The completion script for `shell` ("zsh" or "bash"); throws for any other shell. + static func completionScript(shell: String, commands: [Command]) throws -> String { + let shown = commands.filter { !$0.hidden } + switch shell { + case "zsh": return zshScript(shown) + case "bash": return bashScript(shown) + default: + throw CLIError.usage("unknown shell \"\(shell)\"; supported shells: \(completionShells.joined(separator: ", "))") + } + } + + /// `mlxr completion zsh|bash`: prints the script on stdout. `table` holds the other commands; + /// the script covers them and `completion` itself. + static func completionCommand(table: [Command] = []) -> Command { + let shape = Command(name: "completion", summary: "Generate the shell completion script", args: [Arg("SHELL")]) + var command = shape + command.run = { inv, console in + console.out(try completionScript(shell: inv.positionals[0], commands: table + [shape])) + } + return command + } + + // MARK: - Shared helpers + + private static func sq(_ s: String) -> String { + "'" + s.replacingOccurrences(of: "'", with: "'\\''").replacingOccurrences(of: "\n", with: " ") + "'" + } + + private static func visible(_ c: Command) -> [Opt] { c.options.filter { !$0.hidden } } + + private static func flagNames(_ o: Opt) -> [String] { + ["--" + o.long] + o.aliases.map { "--" + $0 } + (o.short.map { ["-\($0)"] } ?? []) + } + + private static func words(_ k: ValueKind) -> [String]? { + switch k { + case .choice(let w), .choiceOrInt(let w): return w + default: return nil + } + } + + private static func isModel(_ k: ValueKind) -> Bool { if case .model = k { return true } else { return false } } + private static func isFile(_ k: ValueKind) -> Bool { if case .file = k { return true } else { return false } } + + private static let listModels = "command mlxr list 2>/dev/null | awk 'NR>1 {print $1}'" + + // MARK: - zsh + + private static func zshAction(_ k: ValueKind) -> String { + if let w = words(k) { return "(" + w.joined(separator: " ") + ")" } + if isModel(k) { return "_mlxr_models" } + if isFile(k) { return "_files" } + return " " + } + + private static func zshScript(_ commands: [Command]) -> String { + var out = "#compdef mlxr\n\n_mlxr_models() {\n local -a models\n models=(${(f)\"$(\(listModels))\"})\n compadd -a models\n}\n\n" + out += "_mlxr() {\n local -a commands\n commands=(\n" + for c in commands { + for n in [c.name] + c.aliases { out += " " + sq(n + ":" + c.summary.replacingOccurrences(of: "[", with: "(").replacingOccurrences(of: "]", with: ")")) + "\n" } + } + out += " )\n if (( CURRENT == 2 )); then\n _describe 'command' commands\n return\n fi\n case ${words[2]} in\n" + for c in commands { + out += " " + ([c.name] + c.aliases).joined(separator: "|") + ")\n _arguments \\\n" + for o in visible(c) { + let desc = o.help.replacingOccurrences(of: "[", with: "(").replacingOccurrences(of: "]", with: ")") + for n in flagNames(o) { + var spec = n + if o.value != nil, o.implicit != nil, n.hasPrefix("--") { spec += "=-" } + spec += "[\(desc)]" + if let k = o.value { spec += ":value:" + zshAction(k) } + out += " " + sq(spec) + " \\\n" + } + } + out += " " + sq("(-h --help)--help[help]") + " \\\n" + for (i, a) in c.args.enumerated() { + let lead = a.variadic ? "*" : (a.optional ? "\(i + 1):" : "\(i + 1)") + out += " " + sq("\(lead):\(a.name.lowercased()):" + zshAction(a.kind)) + " \\\n" + } + out += " && return\n ;;\n" + } + out += " esac\n}\n\n_mlxr \"$@\"\n" + return out + } + + // MARK: - bash + + private static func bashScript(_ commands: [Command]) -> String { + let names = commands.flatMap { [$0.name] + $0.aliases } + var out = "_mlxr_models() {\n \(listModels)\n}\n\n_mlxr() {\n local cur prev cmd\n COMPREPLY=()\n" + out += " cur=\"${COMP_WORDS[COMP_CWORD]}\"\n prev=\"${COMP_WORDS[COMP_CWORD-1]}\"\n" + out += " if [ \"$COMP_CWORD\" -eq 1 ]; then\n COMPREPLY=($(compgen -W \(sq(names.joined(separator: " "))) -- \"$cur\"))\n return\n fi\n" + out += " cmd=\"${COMP_WORDS[1]}\"\n case \"$cmd\" in\n" + for c in commands { + out += " " + ([c.name] + c.aliases).joined(separator: "|") + ")\n case \"$prev\" in\n" + for o in visible(c) { + guard let k = o.value, o.implicit == nil || words(k) != nil else { continue } + let pattern = flagNames(o).joined(separator: "|") + if let w = words(k) { + out += " \(pattern)) COMPREPLY=($(compgen -W \(sq(w.joined(separator: " "))) -- \"$cur\")); return;;\n" + } else if isModel(k) { + out += " \(pattern)) COMPREPLY=($(compgen -W \"$(_mlxr_models)\" -- \"$cur\")); return;;\n" + } else if isFile(k) { + out += " \(pattern)) COMPREPLY=($(compgen -f -- \"$cur\")); return;;\n" + } else { + out += " \(pattern)) return;;\n" + } + } + out += " esac\n" + let flags = (visible(c).flatMap(flagNames) + ["--help", "-h"]).joined(separator: " ") + out += " if [[ \"$cur\" == -* ]]; then\n COMPREPLY=($(compgen -W \(sq(flags)) -- \"$cur\"))\n return\n fi\n" + if c.args.contains(where: { isModel($0.kind) }) { + out += " COMPREPLY=($(compgen -W \"$(_mlxr_models)\" -- \"$cur\"))\n" + } else if c.args.contains(where: { isFile($0.kind) }) { + out += " COMPREPLY=($(compgen -f -- \"$cur\"))\n" + } + out += " ;;\n" + } + out += " esac\n}\n\ncomplete -F _mlxr mlxr\n" + return out + } +} diff --git a/Sources/MLXRCLI/GenerationEngine.swift b/Sources/MLXRCLI/GenerationEngine.swift new file mode 100644 index 0000000..88f62aa --- /dev/null +++ b/Sources/MLXRCLI/GenerationEngine.swift @@ -0,0 +1,17 @@ +import Foundation +import MLXEngine + +/// The generation engine as the commands use it, injected so tests never touch a GPU or a model. +/// Cancellation does not throw: a cancelled stream ends with `.finished(stats)` and +/// `stats.stopReason == "cancelled"`. +public protocol GenerationEngine: Sendable { + func load(directory: URL, draftDirectory: URL?, contextLength: Int, sparse: SparseAttention?) async throws -> LoadedModelInfo + func chat(_ messages: [ChatMessage], parameters: SamplingParameters, enableThinking: Bool?, + token: CancelToken) -> AsyncThrowingStream + func complete(_ prompt: String, parameters: SamplingParameters, + token: CancelToken) -> AsyncThrowingStream + func renderChat(_ messages: [ChatMessage], enableThinking: Bool?) -> String + func encode(_ text: String) -> [Int] + /// The installed model that pairs with `target` as a draft model, if any (`--draft auto`). + func autoDraft(for target: URL) -> LocalModel? +} diff --git a/Sources/MLXRCLI/Help.swift b/Sources/MLXRCLI/Help.swift new file mode 100644 index 0000000..c1093b2 --- /dev/null +++ b/Sources/MLXRCLI/Help.swift @@ -0,0 +1,67 @@ +import Foundation + +/// Help text in the cobra layout. +extension CLI { + static let rootName = "mlxr" + static let rootSummary = "Run MLX models from Hugging Face on Apple Silicon, from the command line." + + static func rootHelp(commands: [Command]) -> String { + let shown = commands.filter { !$0.hidden } + let width = max(11, shown.map { $0.name.count }.max() ?? 0) + 1 + func rows(_ list: [Command]) -> String { + list.map { " " + $0.name.padding(toLength: width, withPad: " ", startingAt: 0) + " " + $0.summary + "\n" }.joined() + } + var out = rootSummary + "\n\nUsage:\n \(rootName) [flags]\n \(rootName) [command]\n\n" + out += "Available Commands:\n" + rows(shown.filter { $0.group == nil }) + "\n" + var groups: [String] = [] + for c in shown { if let g = c.group, !groups.contains(g) { groups.append(g) } } + for g in groups { out += "\(g) Commands:\n" + rows(shown.filter { $0.group == g }) + "\n" } + out += "Flags:\n -h, --help help for \(rootName)\n -v, --version show version information\n\n" + out += "Use \"\(rootName) [command] --help\" for more information about a command.\n" + return out + } + + static func commandHelp(_ c: Command) -> String { + var out = c.summary + "\n\nUsage:\n \(rootName) \(c.name)" + for a in c.args { + let n = a.variadic ? a.name + "..." : a.name + out += a.optional ? " [\(n)]" : " \(n)" + } + out += " [flags]\n" + if !c.aliases.isEmpty { out += "\nAliases:\n \(([c.name] + c.aliases).joined(separator: ", "))\n" } + if !c.examples.isEmpty { out += "\nExamples:\n" + c.examples.joined(separator: "\n") + "\n" } + var rows: [(String, String)] = c.options.filter { !$0.hidden }.map { o in + let short = o.short.map { "-\($0), " } ?? " " + var left = short + "--" + o.long + if let kind = o.value { left += " " + typeName(kind) } + var help = o.help + if let d = o.defaultText { help += " (default \(d))" } + return (left, help) + } + rows.append(("-h, --help", "help for \(c.name)")) + rows = rows.map { ($0.0.hasPrefix("-") && !$0.0.hasPrefix("--") ? $0.0 : " " + $0.0.trimmingCharacters(in: .init(charactersIn: " ")), $0.1) } + let sorted = rows.sorted { key($0.0) < key($1.0) } + let w = sorted.map { $0.0.count }.max() ?? 0 + out += "\nFlags:\n" + sorted.map { " " + $0.0.padding(toLength: w, withPad: " ", startingAt: 0) + " " + $0.1 + "\n" }.joined() + if !c.environment.isEmpty { + out += "\nEnvironment Variables:\n" + c.environment.map { + " " + $0.name.padding(toLength: 27, withPad: " ", startingAt: 0) + " " + $0.help + "\n" + }.joined() + } + return out + } + + private static func key(_ left: String) -> String { + left.trimmingCharacters(in: .whitespaces).replacingOccurrences(of: "-h, ", with: "").replacingOccurrences(of: "-", with: "") + .lowercased() + } + + static func typeName(_ k: ValueKind) -> String { + switch k { + case .int: return "int" + case .uint64: return "uint" + case .real, .realAboveZero: return "float" + case .text, .choice, .choiceOrInt, .model, .file: return "string" + } + } +} diff --git a/Sources/MLXRCLI/HubServices.swift b/Sources/MLXRCLI/HubServices.swift new file mode 100644 index 0000000..bb7b3b0 --- /dev/null +++ b/Sources/MLXRCLI/HubServices.swift @@ -0,0 +1,36 @@ +import Foundation +import MLXEngine + +/// What a running download reports: bytes done, bytes in total, bytes per second, and status lines. +public struct DownloadHooks: Sendable { + public var progress: @Sendable (_ done: Int64, _ total: Int64, _ bytesPerSecond: Double) -> Void + public var event: @Sendable (String) -> Void + + public init(progress: @escaping @Sendable (Int64, Int64, Double) -> Void, event: @escaping @Sendable (String) -> Void) { + self.progress = progress; self.event = event + } +} + +/// Hugging Face access for pull, show and auto-pull, injected so tests never touch the network or +/// the real model library. +public struct HubServices: Sendable { + public var details: @Sendable (_ repo: String) async throws -> HubModelDetails + public var destination: @Sendable (_ repo: String) throws -> URL + public var download: @Sendable (_ details: HubModelDetails, _ destination: URL, _ hooks: DownloadHooks) async throws -> URL + + public init(details: @escaping @Sendable (String) async throws -> HubModelDetails, + destination: @escaping @Sendable (String) throws -> URL, + download: @escaping @Sendable (HubModelDetails, URL, DownloadHooks) async throws -> URL) { + self.details = details; self.destination = destination; self.download = download + } + + public static let live = HubServices( + details: { try await HubClient.details($0) }, + destination: { try ModelStore.directory(for: $0) }, + download: { details, destination, hooks in + let dl = ModelDownload(details: details, destination: destination) + dl.onProgress = { _, done, total, speed in hooks.progress(done, total, speed) } + dl.onEvent = hooks.event + return try await withTaskCancellationHandler { try await dl.start() } onCancel: { dl.cancel() } + }) +} diff --git a/Sources/MLXRCLI/Interrupts.swift b/Sources/MLXRCLI/Interrupts.swift new file mode 100644 index 0000000..dc217b4 --- /dev/null +++ b/Sources/MLXRCLI/Interrupts.swift @@ -0,0 +1,89 @@ +import Foundation + +/// What the process is doing when Ctrl-C arrives. +public enum InterruptPhase: Equatable, Sendable { case idle, loading, generating, downloading, serving } + +/// What an interrupt asks for. +public enum InterruptAction: Equatable, Sendable { + /// Nothing is running that an interrupt can stop. + case ignore + /// Leave the process now with this code (loading: 130, serving: 0). + case exit(Int32) + /// Cancel the running generation; the command returns by itself (exit 130). + case cancel + /// Cancel the running download, keep the partial files; the command returns by itself (exit 130). + case cancelAndExit(Int32) + /// A second interrupt while a generation is still being cancelled: leave the process at once. + case forceExit(Int32) +} + +/// The Ctrl-C state machine: loading -> exit 130; generating -> cancel, and a second interrupt +/// -> forced exit 130; downloading -> cancel and exit 130; serving -> exit 0. +public struct InterruptMachine: Sendable { + public private(set) var phase: InterruptPhase = .idle + public private(set) var count = 0 + + public init() {} + + /// Starts a phase; the interrupt count starts again. + public mutating func enter(_ phase: InterruptPhase) { self.phase = phase; count = 0 } + + public mutating func interrupt() -> InterruptAction { + count += 1 + switch phase { + case .idle: return .ignore + case .loading: return .exit(130) + case .generating: return count == 1 ? .cancel : .forceExit(130) + case .downloading: return .cancelAndExit(130) + case .serving: return .exit(0) + } + } +} + +/// Set by the cancel action of a phase: true only when the user interrupted (the engine also +/// cancels its token whenever a stream ends, so the token cannot tell). +public final class UserCancel: @unchecked Sendable { + private let lock = NSLock() + private var set = false + public init() {} + public func mark() { lock.lock(); set = true; lock.unlock() } + public var isSet: Bool { lock.lock(); defer { lock.unlock() }; return set } +} + +/// Routes Ctrl-C to the running phase. The live `SIGINT` source (Signals.swift) calls `fire()`; +/// tests call it directly. `exit` is `_exit` in the real process. +public final class Interrupts: @unchecked Sendable { + public static let shared = Interrupts() + + private let lock = NSLock() + private var machine = InterruptMachine() + private var cancelAction: @Sendable () -> Void = {} + private var interrupted = false + private let exitProcess: @Sendable (Int32) -> Void + + public init(exit: @escaping @Sendable (Int32) -> Void = { _exit($0) }) { exitProcess = exit } + + /// Enters a phase. `onCancel` is what `.cancel` / `.cancelAndExit` do (stop the generation, the download). + public func enter(_ phase: InterruptPhase, onCancel: @escaping @Sendable () -> Void = {}) { + lock.lock(); defer { lock.unlock() } + machine.enter(phase) + cancelAction = onCancel + } + + /// An interrupt arrived: applies the state machine (calls `onCancel` or `exit`). + public func fire() { + lock.lock() + let action = machine.interrupt() + let cancel = cancelAction + if action != .ignore { interrupted = true } + lock.unlock() + switch action { + case .ignore: break + case .exit(let c), .forceExit(let c): exitProcess(c) + case .cancel, .cancelAndExit: cancel() + } + } + + /// True once any interrupt arrived. + public var wasInterrupted: Bool { lock.lock(); defer { lock.unlock() }; return interrupted } +} diff --git a/Sources/MLXRCLI/Legacy.swift b/Sources/MLXRCLI/Legacy.swift new file mode 100644 index 0000000..c4c6938 --- /dev/null +++ b/Sources/MLXRCLI/Legacy.swift @@ -0,0 +1,427 @@ +import Foundation +import MLXEngine + +// The original `mlxr` command bodies, kept as they were while the commands move onto the parser kit. + +/// `--draft `: the draft model for speculative decoding. +func draftDirectory(_ s: String?, target: URL) throws -> URL? { + guard let s else { return nil } + guard s == "auto" else { return try installedModelDirectory(s, resolver: .live) } + guard let m = InferenceEngine.autoDraft(for: target) else { + print("no installed draft model pairs with \(target.lastPathComponent); running without one") + return nil + } + return m.directory +} + +func sparseSummary(_ info: LoadedModelInfo) -> String { + if info.sparseLayers > 0 { + return "sparse attention on \(info.sparseLayers) layers from \(info.sparseMinContext) tokens (\(info.sparseTopK) long-range keys)" + } + return "sparse attention off" + (info.sparseNote.map { ": \($0)" } ?? "") +} + +func formatBytes(_ b: Int) -> String { + ByteCountFormatter.string(fromByteCount: Int64(b), countStyle: .file) +} + +/// Long-context settings of `mlxr bench` (MLXR_BENCH_CTX turns the section on). +struct LongBench { + let contexts: [Int] + /// Teacher-forced positions of the text after the prompt (MLXR_BENCH_POSITIONS, default 512). + let positions: Int + /// Token offsets into the text where prompts start (MLXR_BENCH_TEXT_OFFSETS, default 0). + let offsets: [Int] + static let steps = 64 + /// Rows the model needs past the longest prompt: the measured positions plus a needle question. + var margin: Int { max(Self.steps, positions) + 96 } + + init(env: [String: String]) { + func ints(_ key: String) -> [Int] { + (env[key] ?? "").split(separator: ",").compactMap { Int($0.trimmingCharacters(in: .whitespaces)) } + } + contexts = ints("MLXR_BENCH_CTX").filter { $0 > 0 } + positions = min(max(ints("MLXR_BENCH_POSITIONS").first ?? 512, 0), 8192) + let o = ints("MLXR_BENCH_TEXT_OFFSETS").filter { $0 >= 0 } + offsets = o.isEmpty ? [0] : o + } +} + +/// `mlxr bench` with MLXR_BENCH_CTX: decode after long prompts, dense and (for a model loaded with +/// MLXR_SPARSE_ATTN) sparse. For each context: +/// - greedy decode speed, and how many greedy tokens sparse decode shares with dense decode; +/// - teacher-forced on dense decode's tokens: a control (dense forward against the GPU's greedy +/// tokens, must be all) and sparse argmax agreement; +/// - teacher-forced on the text after the prompt (MLXR_BENCH_TEXT only): KL(dense‖sparse), the +/// text's NLL under both, and top-1 agreement by dense decode's top-1 margin. Each position runs +/// sparse first and then dense, so the cache stays dense decode's; +/// - the cost of a dense verification pass of T tokens at that context; +/// - needles (MLXR_BENCH_TEXT only): five passkeys at 10–90% depth, recalled dense and sparse. +/// The prompt is MLXR_BENCH_TEXT from each of MLXR_BENCH_TEXT_OFFSETS, or pseudo-random tokens, on +/// which agreement says little. +func benchLongContext(_ t: Transformer, dir: URL, bench: LongBench) throws { + var greedy = SamplingParameters() + greedy.temperature = 0 + greedy.repetitionPenalty = 1 + let sp = SampleParams(greedy, vocab: t.vocabSize) + let V = t.vocabSize + let tokenizer = try Tokenizer(directory: dir) + var text: [Int32] = [] + if let path = ProcessInfo.processInfo.environment["MLXR_BENCH_TEXT"] { + guard let s = try? String(contentsOfFile: path, encoding: .utf8) else { throw CLIError.failure("cannot read MLXR_BENCH_TEXT \(path)") } + text = tokenizer.encode(s).map { Int32($0) }.filter { $0 >= 0 && Int($0) < V } + guard !text.isEmpty else { throw CLIError.failure("MLXR_BENCH_TEXT \(path) has no tokens") } + let bos = tokenizer.bosToken.flatMap { b in tokenizer.tokenId(b).map { " (prompts start with \(b), id \($0))" } } ?? "" + print("long-context prompt: \(path), \(text.count) tokens (repeated to length)\(bos)") + } else { + print("long-context prompt: pseudo-random tokens (MLXR_BENCH_TEXT= for real text)") + } + let original = t.sparse + defer { t.sparse = original } + // Hybrid models (recurrent state): every rerun of a position first restores the state there. + var base: Transformer.RecurrentSnapshot?, last: Transformer.RecurrentSnapshot? + func mark() { if t.config.isHybrid { last = t.snapshotRecurrent() } } + func seek(_ pos: Int) throws { + guard t.config.isHybrid, t.recurrentPosition != pos else { return } + guard let s = [base, last].compactMap({ $0 }).first(where: { $0.position == pos }) else { throw CLIError.failure("no recurrent snapshot at \(pos)") } + t.restoreRecurrent(s) + } + let sparse = t.sparseAvailable ? original : nil + let steps = LongBench.steps + func logits(_ tokens: [Int32], offset: Int) throws -> [Float] { + Array(UnsafeBufferPointer(start: try t.forward(tokens, offset: offset), count: V)) + } + func argmax(_ x: [Float]) -> Int { + var best = 0 + for v in 1.. x[best] { best = v } + return best + } + func logSoftmax(_ x: [Float]) -> [Double] { + let m = Double(x.max() ?? 0) + var s = 0.0 + for v in x { s += exp(Double(v) - m) } + let lse = m + log(s) + return x.map { Double($0) - lse } + } + for ctx in bench.contexts { + guard ctx + bench.margin < t.maxSeq else { print("decode @\(ctx): skipped (the model's context is \(t.maxSeq))"); continue } + for (oi, off) in (text.isEmpty ? [0] : bench.offsets).enumerated() { + var rng: UInt64 = 0x9E3779B97F4A7C15 + let lo = min(1000, V / 2), span = UInt64(max(1, min(V, 30000) - lo)) + func token(_ i: Int) -> Int32 { + if !text.isEmpty { return text[(off + i) % text.count] } + rng ^= rng << 13; rng ^= rng >> 7; rng ^= rng << 17 + return Int32(lo + Int(rng % span)) + } + var prompt = (0..= 0, id < V { prompt[0] = Int32(id) } + } + let at = off > 0 ? " from text token \(off)" : "" + t.sparse = nil + t.seedSampler(1) + let s0 = Date() + let first = try t.prefill(prompt, offset: 0, sample: sp) + let prefillTime = Date().timeIntervalSince(s0) + if t.config.isHybrid { base = t.snapshotRecurrent() } + print(String(format: "prefill @%d%@: %.1f s (%.0f tok/s)", ctx, at as NSString, prefillTime, Double(ctx) / prefillTime)) + // Each run decodes from the same prompt (positions from ctx on are rewritten, the index with them). + var dense: [Int] = [] + var modes: [(String, SparseAttention?)] = [("dense", nil)] + if let sparse, oi == 0 { modes.append(("sparse", sparse)) } + for (label, mode) in modes { + t.sparse = mode + t.seedSampler(1) + try seek(ctx) + var tokens: [Int] = [] + var gpu = 0.0 + let s = Date() + var cur = try t.beginDecodeStep(offset: ctx, sample: sp) + for i in 0.. 0 { + // Teacher-forced on the text after the prompt. + let N = bench.positions + var kl: [Double] = [], nllDense = 0.0, nllSparse = 0.0, agree = 0 + let edges: [Float] = [0.1, 0.5, 1] + var bucketTotal = [Int](repeating: 0, count: edges.count + 1), bucketAgree = bucketTotal + try seek(ctx) + for i in 0.. -40 { k += exp(pd[v]) * (pd[v] - ps[v]) } + kl.append(max(0, k)) + nllDense -= pd[target] + nllSparse -= ps[target] + let top = argmax(ld) + var second = top == 0 ? 1 : 0 + for v in 0.. ld[second] { second = v } + let bucket = edges.firstIndex { ld[top] - ld[second] < $0 } ?? edges.count + bucketTotal[bucket] += 1 + if argmax(ls) == top { agree += 1; bucketAgree[bucket] += 1 } + } + kl.sort() + let meanKL = kl.reduce(0, +) / Double(N) + let dNLL = (nllSparse - nllDense) / Double(N) + print(String(format: " teacher-forced on the text (%d positions): KL(dense‖sparse) mean %.4f, median %.4f, p95 %.4f, max %.3f nats", + N, meanKL, kl[N / 2], kl[min(N - 1, N * 95 / 100)], kl[N - 1])) + print(String(format: " text NLL: dense %.4f, sparse %.4f nats/token (Δ %+.4f, perplexity ×%.4f)", + nllDense / Double(N), nllSparse / Double(N), dNLL, exp(dNLL))) + let labels = ["<0.1", "0.1–0.5", "0.5–1", "≥1"] + let byMargin = zip(labels, zip(bucketAgree, bucketTotal)).map { "\($0.0): \($0.1.0)/\($0.1.1)" }.joined(separator: ", ") + print(String(format: " top-1 agreement %d/%d (%.1f%%); by dense top-1 margin (nats) %@", agree, N, + 100 * Double(agree) / Double(N), byMargin as NSString)) + } + if oi == 0 && !t.config.isHybrid { + // A dense verification pass of T tokens at this context (speculative decoding). + t.sparse = nil + var line: [String] = [] + for T in [1, 2, 5, 8] where T <= Transformer.maxVerifyTokens { + var gpu = 0.0 + for i in 0..<4 { + _ = try t.forward((0.. Bool { + lock.lock(); defer { lock.unlock() } + guard Date().timeIntervalSince(date) >= s else { return false } + date = Date() + return true + } +} + +func legacySearch(_ query: String) async throws { + let results = try await HubClient.search(query, limit: 25) + for m in results { + print(String(format: "%-60@ %10d ↓ %@", m.id as NSString, m.downloads ?? 0, (m.modelTypeTag ?? "") as NSString)) + } + +} + +func legacyDownload(_ repo: String) async throws { + try validateRepoID(repo) + let details = try await HubClient.details(repo) + if !details.isSupported { print("warning: \(details.modelType ?? "unknown") architecture is not supported by the engine") } + print("Downloading \(repo) (\(formatBytes(details.downloadSize)))") + let dl = ModelDownload(details: details, destination: try ModelStore.directory(for: repo)) + let last = LockedDate() + dl.onProgress = { p, done, total, speed in + guard last.elapsed(0.25) || p >= 1 else { return } + print(String(format: "\r %5.1f%% %@ / %@ %@/s ", p * 100, formatBytes(Int(done)), formatBytes(Int(total)), formatBytes(Int(speed))), terminator: "") + } + dl.onEvent = { print("\n \($0)") } + let dir = try await performDownload(start: { try await dl.start() }) + print("\nSaved to \(dir.path)") + +} + +func legacyTokenize(_ model: String, _ text: String, idsOnly: Bool = false) throws { + let tok = try Tokenizer(directory: try installedModelDirectory(model, resolver: .live)) + let ids = tok.encode(text) + print(ids) + if idsOnly { return } + print(ids.map { tok.token($0) ?? "?" }) + print("roundtrip:", tok.decode(ids, includeSpecial: true)) + +} + +func legacyBench(_ model: String) throws { + let dir = try installedModelDirectory(model, resolver: .live) + // Long-context decode (opt-in): MLXR_BENCH_CTX=8192,32768 adds a section at the end. + let longBench = LongBench(env: ProcessInfo.processInfo.environment) + let t = try Transformer(directory: dir, contextLength: max(4096, (longBench.contexts.max() ?? 0) + longBench.margin + 64)) + print("weights: \(formatBytes(t.weightBytes))") + if let sp = t.sparse { + print(t.sparseAvailable + ? "sparse decode attention: top-k \(sp.topK), window \(sp.window), sink \(sp.sink), from \(sp.minContext) tokens" + : "sparse decode attention: off (\(t.sparseOffReason ?? "unavailable"))") + } + print("attention: \(t.attentionSummary)") + let quick = ProcessInfo.processInfo.environment["MLXR_BENCH_DECODE"] != nil + let noProfile = quick || ProcessInfo.processInfo.environment["MLXR_BENCH_NOPROFILE"] != nil + if !noProfile { print(try t.profile()) } + if ProcessInfo.processInfo.environment["MLXR_BENCH_PROFILE_ONLY"] != nil { exit(0) } + _ = try t.forward([1], offset: 0) + for n in (quick ? [] : [128, 512, 2048]) { + let toks = (0.. 0 { + print(String(format: " sampler: %.2f attempts and pool %.0f per token on average, at most %d attempts", Double(diag.attempts) / Double(steps + 1), Double(diag.pool) / Double(steps + 1), diag.maxAttempts)) + } + } + } + // Speculative verification cost: one forward over T tokens with logits for every row. + // At a short context and after a 1900-token prompt (the attention part grows with the context). + // (A recurrent state cannot rewind to rerun a position: no verification on hybrid models.) + for ctx in t.config.isHybrid ? [] : [64, 1900] { + if ctx > 64 { _ = try t.forward((0.. 64 ? "verify @\(ctx)" : "verify" + print(String(format: "%@ T=%d: %6.2f ms per pass (%5.2f ms per token, gpu %.2f ms)", label as NSString, T, dt * 1000, dt * 1000 / Double(T), gpu / Double(steps) * 1000)) + } + } + for T in [5, 8, 12, 15] { + let s = Date() + for i in 0..<8 { _ = try t.forward((0.. String? { + guard let key else { return nil } + return generated ? "API key (send as 'Authorization: Bearer '): \(key)" + : "API key required: send it as 'Authorization: Bearer '" +} + +func legacyServe(model: String, port: UInt16, ctx: Int, lan: Bool, apiKey: String?, draft: String?, sparse: String?) async throws { + let sparseArg = try sparseSetting(sparse) + // LAN mode always needs a key: use --api-key / MLXR_API_KEY, or generate one. + var key = apiKey ?? ProcessInfo.processInfo.environment["MLXR_API_KEY"] + let generated = lan && key == nil + if generated { key = OpenAIServer.generateAPIKey() } + let dir = try installedModelDirectory(model, resolver: .live) + let info = try await InferenceEngine.shared.load(directory: dir, draftDirectory: try draftDirectory(draft, target: dir), contextLength: ctx, sparse: sparseArg) + print("Loaded \(info.name)\(info.draftName.map { " + draft \($0)" } ?? "") in \(String(format: "%.2f", info.loadSeconds))s") + print(sparseSummary(info)) + let server = OpenAIServer(port: port, allowLAN: lan, apiKey: key) + server.onLog = { print($0) } + try startServer { try server.start() } + print("OpenAI-compatible server on http://\(lan ? "0.0.0.0" : "127.0.0.1"):\(port)/v1 (Ctrl-C to stop)") + if let notice = serveKeyNotice(key: key, generated: generated) { print(notice) } + while true { try await Task.sleep(nanoseconds: 3_600_000_000_000) } +} diff --git a/Sources/MLXRCLI/Library.swift b/Sources/MLXRCLI/Library.swift new file mode 100644 index 0000000..404e165 --- /dev/null +++ b/Sources/MLXRCLI/Library.swift @@ -0,0 +1,206 @@ +import CryptoKit +import Foundation +import MLXEngine + +/// The installed models, injected so tests never scan or delete the real library. +public struct ModelLibrary: Sendable { + public var scan: @Sendable () -> [LocalModel] + public var delete: @Sendable (LocalModel) throws -> Void + + public init(scan: @escaping @Sendable () -> [LocalModel], delete: @escaping @Sendable (LocalModel) throws -> Void) { + self.scan = scan; self.delete = delete + } + + public static let live = ModelLibrary(scan: { ModelStore.scan() }, delete: { try ModelStore.delete($0) }) +} + +/// A named alias for a model, with the system prompt and parameters saved alongside it. +public struct Preset: Codable, Equatable, Sendable { + public var name: String + public var repo: String + public var system: String? + public var parameters: [String: String] + + public init(name: String, repo: String, system: String? = nil, parameters: [String: String] = [:]) { + self.name = name; self.repo = repo; self.system = system; self.parameters = parameters + } +} + +/// Presets as one `/.json` file each, read and written safely. +public struct PresetStore: Sendable { + public var directory: URL + + public init(directory: URL) { self.directory = directory } + + public static let live = PresetStore(directory: ModelStore.root.deletingLastPathComponent().appendingPathComponent("presets", isDirectory: true)) + + /// A preset name is 1...128 characters of letters, digits and `. _ - :`, not starting with a dot. + public static func validateName(_ name: String) throws { + let ok = !name.isEmpty && name.utf8.count <= 128 && !name.hasPrefix(".") && name.utf8.allSatisfy { + ($0 >= 48 && $0 <= 58) || ($0 >= 65 && $0 <= 90) || ($0 >= 97 && $0 <= 122) || $0 == 46 || $0 == 95 || $0 == 45 + } + guard ok else { throw CLIError.failure("invalid model name '\(name)' (use letters, digits and . _ - :)") } + } + + private func file(_ name: String) -> URL { directory.appendingPathComponent(name + ".json") } + + /// The preset, or nil when missing, unreadable (moved aside) or the name is unsafe. + public func load(_ name: String) -> Preset? { + guard (try? Self.validateName(name)) != nil else { return nil } + if case .loaded(let p) = SafeJSONFile.load(Preset.self, from: file(name)) { return p } + return nil + } + + public func list() -> [Preset] { + let names = ((try? FileManager.default.contentsOfDirectory(atPath: directory.path)) ?? []) + .filter { $0.hasSuffix(".json") && !$0.contains(".corrupt-") }.map { String($0.dropLast(5)) } + return names.compactMap { load($0) }.sorted { $0.name < $1.name } + } + + public func save(_ preset: Preset) throws { + try Self.validateName(preset.name) + try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) + let encoder = JSONEncoder() + encoder.outputFormatting = [.prettyPrinted, .sortedKeys] + try encoder.encode(preset).write(to: file(preset.name), options: .atomic) + } + + public func delete(_ name: String) throws { + try Self.validateName(name) + let url = file(name) + if FileManager.default.fileExists(atPath: url.path) { try FileManager.default.removeItem(at: url) } + } +} + +/// What a model name resolved to. +public struct Resolution: Equatable, Sendable { + public enum Source: String, Sendable { case path, installed, huggingFace, catalog, preset, shortName } + public var source: Source + /// The Hugging Face repo id; nil for a plain path. + public var repo: String? + /// Where the model is on disk; nil when it is not installed. + public var directory: URL? + /// The preset a name resolved through. + public var preset: Preset? + + public init(source: Source, repo: String?, directory: URL?, preset: Preset? = nil) { + self.source = source; self.repo = repo; self.directory = directory; self.preset = preset + } +} + +/// Turns what the user typed into a model: path, installed repo id, `hf.co/org/repo`, catalog alias, +/// preset, installed short name, in that order. +public struct ModelResolver: Sendable { + public var library: ModelLibrary + public var presets: PresetStore + public var catalog: Catalog + + public init(library: ModelLibrary, presets: PresetStore, catalog: Catalog = .standard) { + self.library = library; self.presets = presets; self.catalog = catalog + } + + public func resolve(_ name: String) throws -> Resolution { + guard !name.isEmpty else { throw CLIError.failure("model name is empty") } + if FileManager.default.fileExists(atPath: name) { + return Resolution(source: .path, repo: nil, directory: URL(fileURLWithPath: name)) + } + let models = library.scan() + func installed(_ repo: String) -> LocalModel? { models.first { $0.repo.lowercased() == repo.lowercased() } } + func make(_ source: Resolution.Source, _ repo: String, preset: Preset? = nil) -> Resolution { + let m = installed(repo) + return Resolution(source: source, repo: m?.repo ?? repo, directory: m?.directory, preset: preset) + } + if let m = installed(name) { return make(.installed, m.repo) } + let lower = name.lowercased() + for prefix in ["hf.co/", "huggingface.co/"] where lower.hasPrefix(prefix) { + let repo = String(name.dropFirst(prefix.count)) + try validateRepo(repo) + return make(.huggingFace, repo) + } + if let repo = catalog.repo(forAlias: name) { return make(.catalog, repo) } + if let p = presets.load(name) { + try validateRepo(p.repo) + return make(.preset, p.repo, preset: p) + } + let named = models.filter { $0.name.lowercased() == lower } + if named.count > 1 { + throw CLIError.failure("'\(name)' is ambiguous; matches: " + named.map(\.repo).joined(separator: ", ")) + } + if let m = named.first { return make(.shortName, m.repo) } + if name.contains("/") { + try validateRepo(name) + return make(.huggingFace, name) + } + throw CLIError.failure(missMessage(name, models: models)) + } + + private func validateRepo(_ repo: String) throws { + guard RepoID.isValid(repo) else { throw CLIError.failure("invalid repo id: \(repo) (expected owner/name)") } + } + + private func missMessage(_ name: String, models: [LocalModel]) -> String { + let lower = name.lowercased() + var candidates: [(shown: String, keys: [String])] = catalog.aliases.map { ($0, [$0]) } + candidates += models.map { ($0.repo, [$0.repo.lowercased(), $0.name.lowercased()]) } + candidates += presets.list().map { ($0.name, [$0.name.lowercased()]) } + var scored: [(String, Int)] = [] + for c in candidates where !scored.contains(where: { $0.0 == c.shown }) { + let d = c.keys.map { CLI.editDistance(lower, $0) }.min() ?? 99 + if d <= 2 { scored.append((c.shown, d)) } + } + var msg = "model '\(name)' not found" + let best = scored.sorted { ($0.1, $0.0) < ($1.1, $1.0) }.prefix(3).map { "'\($0.0)'" } + if !best.isEmpty { msg += ", did you mean " + best.joined(separator: " or ") + "?" } + return msg + } +} + +extension CLI { + /// Base-1000 size: `%d UNIT` from 10, else `%.1f UNIT` (`5.2 GB`, `350 MB`). + static func humanBytes(_ n: Int) -> String { + if n < 1000 { return "\(max(n, 0)) B" } + var value = Double(n) / 1000 + var unit = "KB" + for next in ["MB", "GB", "TB"] where value >= 1000 { value /= 1000; unit = next } + return (value >= 10 ? String(Int(value)) : String(format: "%.1f", value)) + " " + unit + } + + /// Ollama's relative time (`Less than a second`, `About a minute`, `5 minutes`, `3 weeks`) plus ` ago`. + static func relativeTime(_ date: Date, now: Date) -> String { + let delta = now.timeIntervalSince(date) + let seconds = Int(abs(delta)) + let hours = seconds / 3600 + let text: String + switch seconds { + case 0: text = "Less than a second" + case 1: text = "1 second" + case ..<60: text = "\(seconds) seconds" + case ..<120: text = "About a minute" + case ..<3600: text = "\(seconds / 60) minutes" + default: + if hours == 1 { text = "About an hour" } + else if hours < 48 { text = "\(hours) hours" } + else if hours < 24 * 14 { text = "\(hours / 24) days" } + else if hours < 24 * 60 { text = "\(hours / 24 / 7) weeks" } + else if hours < 24 * 365 * 2 { text = "\(hours / 24 / 30) months" } + else { text = "\(hours / 24 / 365) years" } + } + return text + (delta < 0 ? " from now" : " ago") + } + + /// Borderless table: every column but the last padded to its widest cell, then 4 spaces; the last + /// column's body cells followed by 4 spaces, its header by 1. + static func renderTable(header: [String], rows: [[String]]) -> String { + let last = header.count - 1 + let widths = header.indices.map { i in ([header[i]] + rows.map { $0[i] }).map(\.count).max() ?? 0 } + func line(_ cells: [String], tail: String) -> String { + cells.indices.map { i in i == last ? cells[i] + tail : cells[i].padding(toLength: widths[i], withPad: " ", startingAt: 0) + " " }.joined() + "\n" + } + return line(header, tail: " ") + rows.map { line($0, tail: " ") }.joined() + } + + /// The 12 hex characters shown as ID: the start of the SHA-256 of the repo id. + static func modelID(_ repo: String) -> String { + SHA256.hash(data: Data(repo.utf8)).prefix(6).map { String(format: "%02x", $0) }.joined() + } +} diff --git a/Sources/MLXRCLI/ModelCommands.swift b/Sources/MLXRCLI/ModelCommands.swift new file mode 100644 index 0000000..97c33d3 --- /dev/null +++ b/Sources/MLXRCLI/ModelCommands.swift @@ -0,0 +1,128 @@ +import Foundation +import MLXEngine + +extension CLI { + static let notSupported = ["create", "push", "signin", "signout", "launch"] + + /// list, rm, cp, ps, stop and the commands mlxr does not have. + static func modelCommands(library: ModelLibrary, presets: PresetStore, catalog: Catalog, + now: @escaping @Sendable () -> Date) -> [Command] { + let resolver = ModelResolver(library: library, presets: presets, catalog: catalog) + return [ + Command(name: "list", aliases: ["ls"], summary: "List installed models", args: [Arg("PREFIX", optional: true)], + options: [Opt(long: "json", help: "print the models as JSON")], + run: { inv, console in + console.out(renderList(library: library, presets: presets, prefix: inv.positionals.first, + json: inv.bool("json"), now: now())) + }), + Command(name: "rm", summary: "Remove a model", args: [Arg("MODEL", kind: .model, variadic: true)], + options: [Opt(long: "yes", short: "y", help: "do not ask before deleting from the Hugging Face cache")], + run: { inv, console in try remove(inv, console, library: library, presets: presets, resolver: resolver) }), + Command(name: "cp", summary: "Copy a model", args: [Arg("SOURCE", kind: .model), Arg("DESTINATION")], + run: { inv, console in + let dest = inv.positionals[1] + try PresetStore.validateName(dest) + let source = inv.positionals[0] + let r = try resolver.resolve(source) + guard let repo = r.repo else { throw CLIError.failure("cannot copy a model path; use a model name") } + try presets.save(Preset(name: dest, repo: repo, system: r.preset?.system, parameters: r.preset?.parameters ?? [:])) + console.out("copied '\(source)' to '\(dest)'\n") + console.err("success\n") + }), + Command(name: "ps", summary: "List running models", args: [Arg("PREFIX", optional: true)], + run: { _, console in + console.out(renderTable(header: ["NAME", "ID", "SIZE", "PROCESSOR", "CONTEXT", "RUNNER", "UNTIL"], rows: [])) + }), + Command(name: "stop", summary: "Stop a running model", args: [Arg("MODEL", kind: .model)], + run: { inv, _ in + throw CLIError.failure("couldn't find model \"\(inv.positionals[0])\" to stop\n" + + "mlxr runs a model only while a command uses it and keeps no background server; use 'mlxr serve MODEL' to start one") + }), + ] + unsupportedCommands + } + + /// create, push, signin, signout and launch exist in Ollama only. + static var unsupportedCommands: [Command] { + func stub(_ name: String, aliases: [String] = [], summary: String, options: [Opt] = []) -> Command { + let hint = name == "create" ? "; use 'mlxr pull '" : "" + return Command(name: name, aliases: aliases, summary: summary, args: [Arg("ARGS", optional: true, variadic: true)], + options: options, run: { _, _ in throw CLIError.failure("'\(name)' is not supported by mlxr\(hint)") }) + } + return [ + stub("create", summary: "Create a model (not supported)", options: [ + Opt(long: "file", short: "f", value: .text, help: "Modelfile"), Opt(long: "quantize", short: "q", value: .text, help: "quantization"), + Opt(long: "combine", help: "combine"), Opt(long: "draft-quantize", value: .text, help: "draft quantization"), + Opt(long: "force", help: "force")]), + stub("push", summary: "Push a model (not supported)", options: [Opt(long: "insecure", help: "use an insecure registry")]), + stub("signin", aliases: ["login"], summary: "Sign in (not supported)"), + stub("signout", aliases: ["logout"], summary: "Sign out (not supported)"), + stub("launch", summary: "Launch an integration (not supported)", options: [ + Opt(long: "model", short: "m", value: .text, help: "model"), Opt(long: "config", help: "config"), + Opt(long: "restore", help: "restore"), Opt(long: "yes", short: "y", help: "assume yes")]), + ] + } + + private static func renderList(library: ModelLibrary, presets: PresetStore, prefix: String?, json: Bool, now: Date) -> String { + struct Row { var name: String; var id: String; var size: Int?; var modified: Date? } + let models = library.scan() + var rows = models.map { Row(name: $0.repo, id: modelID($0.repo), size: $0.sizeBytes, modified: $0.modified) } + for p in presets.list() { + let base = models.first { $0.repo.lowercased() == p.repo.lowercased() } + rows.append(Row(name: p.name, id: modelID(p.repo), size: base?.sizeBytes, modified: nil)) + } + if let prefix = prefix?.lowercased(), !prefix.isEmpty { + rows = rows.filter { r in + r.name.lowercased().hasPrefix(prefix) || (r.name.split(separator: "/").last.map { $0.lowercased().hasPrefix(prefix) } ?? false) + } + } + if json { + let iso = ISO8601DateFormatter() + let items: [[String: Any]] = rows.map { r in + var d: [String: Any] = ["name": r.name, "id": r.id, "size": r.size ?? 0] + if let m = r.modified { d["modified"] = iso.string(from: m) } + return d + } + let data = (try? JSONSerialization.data(withJSONObject: items, options: [.prettyPrinted, .sortedKeys])) ?? Data("[]".utf8) + return String(decoding: data, as: UTF8.self) + "\n" + } + return renderTable(header: ["NAME", "ID", "SIZE", "MODIFIED"], rows: rows.map { + [$0.name, $0.id, $0.size.map(humanBytes) ?? "-", $0.modified.map { relativeTime($0, now: now) } ?? "-"] + }) + } + + private enum Removal { case preset(String), model(LocalModel) } + + private static func remove(_ inv: Invocation, _ console: Console, library: ModelLibrary, presets: PresetStore, + resolver: ModelResolver) throws { + let models = library.scan() + var plan: [(arg: String, what: Removal)] = [] + var seen = Set() + for arg in inv.positionals { + if presets.load(arg) != nil { + if seen.insert("preset:" + arg).inserted { plan.append((arg, .preset(arg))) } + continue + } + let r = try resolver.resolve(arg) + guard let repo = r.repo, let m = models.first(where: { $0.repo.lowercased() == repo.lowercased() }) else { + throw CLIError.failure("model '\(arg)' not found") + } + if seen.insert("model:" + m.repo).inserted { plan.append((arg, .model(m))) } + } + if !inv.bool("yes") && console.stdinIsTTY && console.stderrIsTTY { + for (arg, what) in plan { + guard case .model(let m) = what, m.source == .huggingFaceCache else { continue } + console.err("Delete '\(arg)' from the shared Hugging Face cache? [y/N] ") + guard let answer = console.readLine()?.trimmingCharacters(in: .whitespaces).lowercased(), ["y", "yes"].contains(answer) else { + throw CLIError.failure("aborted") + } + } + } + for (arg, what) in plan { + switch what { + case .preset(let name): try presets.delete(name) + case .model(let m): try library.delete(m) + } + console.out("deleted '\(arg)'\n") + } + } +} diff --git a/Sources/MLXRCLI/Parser.swift b/Sources/MLXRCLI/Parser.swift new file mode 100644 index 0000000..0663c41 --- /dev/null +++ b/Sources/MLXRCLI/Parser.swift @@ -0,0 +1,132 @@ +import Foundation + +/// What an option or positional argument accepts. Drives validation, help and completions. +public enum ValueKind { + case text + case int(ClosedRange) + case uint64 + case real(ClosedRange) + /// A finite number greater than 0 and at most `max`. + case realAboveZero(max: Double) + case choice([String]) + case choiceOrInt([String]) + case model + case file +} + +/// One option of a command. `value == nil` is a boolean flag (`--x`, `--x=true|false`). +public struct Opt { + public var long: String + public var aliases: [String] = [] + public var short: Character? = nil + public var value: ValueKind? = nil + /// Value taken when the option is given alone (cobra NoOptDefVal): `--think` means `--think=true`. + public var implicit: String? = nil + public var repeatable = false + public var env: String? = nil + public var defaultText: String? = nil + public var hidden = false + public var help: String = "" + + public init(long: String, aliases: [String] = [], short: Character? = nil, value: ValueKind? = nil, + implicit: String? = nil, repeatable: Bool = false, env: String? = nil, + defaultText: String? = nil, hidden: Bool = false, help: String = "") { + self.long = long; self.aliases = aliases; self.short = short; self.value = value + self.implicit = implicit; self.repeatable = repeatable; self.env = env + self.defaultText = defaultText; self.hidden = hidden; self.help = help + } +} + +/// One positional argument. A variadic argument takes all remaining positionals. +public struct Arg { + public var name: String + public var kind: ValueKind = .text + public var optional = false + public var variadic = false + + public init(_ name: String, kind: ValueKind = .text, optional: Bool = false, variadic: Bool = false) { + self.name = name; self.kind = kind; self.optional = optional; self.variadic = variadic + } +} + +/// Failures a command can report. `usage` and `failure` print `Error: ` and exit 1. +public enum CLIError: Error, Equatable { + case usage(String) + case failure(String) + case interrupted + /// Already reported on stderr: exit 1 without another message. + case reported +} + +/// The streams and terminal facts a command sees; injected so commands are testable. +public final class Console: @unchecked Sendable { + public var out: (String) -> Void + public var err: (String) -> Void + public var stdoutIsTTY: Bool + public var stderrIsTTY: Bool + public var stdinIsTTY: Bool + public var width: Int + public var env: [String: String] + /// Reads one line of input (a confirmation answer); nil at end of input. + public var readLine: () -> String? = { Swift.readLine() } + + public init(out: @escaping (String) -> Void, err: @escaping (String) -> Void, + stdoutIsTTY: Bool = false, stderrIsTTY: Bool = false, stdinIsTTY: Bool = false, + width: Int = 80, env: [String: String] = [:]) { + self.out = out; self.err = err + self.stdoutIsTTY = stdoutIsTTY; self.stderrIsTTY = stderrIsTTY; self.stdinIsTTY = stdinIsTTY + self.width = width; self.env = env + } + + /// Reads everything piped to stdin (only called when stdin is not a terminal). + public var readAllStdin: () -> String = { String(decoding: FileHandle.standardInput.readDataToEndOfFile(), as: UTF8.self) } +} + +/// The validated result of parsing one command line. +public struct Invocation { + public var command: Command + public var positionals: [String] + /// Raw validated values by long option name, in the order given. + public var values: [String: [String]] + + public init(command: Command, positionals: [String] = [], values: [String: [String]] = [:]) { + self.command = command; self.positionals = positionals; self.values = values + } + + /// True when `-h` / `--help` was given: the command does not run. + public var helpRequested = false + + /// Last value given for the option, or nil. + public func string(_ long: String) -> String? { values[long]?.last } + public func strings(_ long: String) -> [String] { values[long] ?? [] } + public func int(_ long: String) -> Int? { string(long).flatMap { Int($0) } } + public func double(_ long: String) -> Double? { string(long).flatMap { Double($0) } } + /// A boolean flag: true when given alone or as `=true`, false for `=false` or when absent. + public func bool(_ long: String) -> Bool { string(long).map { $0 != "false" } ?? false } + public func has(_ long: String) -> Bool { values[long] != nil } +} + +public struct Command { + public var name: String + public var aliases: [String] = [] + public var summary: String + public var args: [Arg] = [] + public var options: [Opt] = [] + public var examples: [String] = [] + public var environment: [(name: String, help: String)] = [] + /// Help heading in the root command list; nil means the main list. + public var group: String? = nil + /// Report a missing variadic argument as `accepts N arg(s), received M` instead of `requires at least`. + public var exactArgWording = false + public var hidden = false + public var run: (Invocation, Console) async throws -> Void + + public init(name: String, aliases: [String] = [], summary: String, args: [Arg] = [], + options: [Opt] = [], examples: [String] = [], + environment: [(name: String, help: String)] = [], group: String? = nil, + hidden: Bool = false, exactArgWording: Bool = false, run: @escaping (Invocation, Console) async throws -> Void = { _, _ in }) { + self.name = name; self.aliases = aliases; self.summary = summary; self.args = args + self.options = options; self.examples = examples; self.environment = environment + self.group = group; self.hidden = hidden; self.exactArgWording = exactArgWording; self.run = run + } +} diff --git a/Sources/MLXRCLI/Parsing.swift b/Sources/MLXRCLI/Parsing.swift new file mode 100644 index 0000000..7f6e708 --- /dev/null +++ b/Sources/MLXRCLI/Parsing.swift @@ -0,0 +1,130 @@ +import Foundation + +/// The command-line grammar of cobra/pflag: `--x v`, `--x=v`, `-x v`, `--` ends options, +/// options and positionals mix freely, the last value of an option wins. +extension CLI { + static func findCommand(_ name: String, in commands: [Command]) -> Command? { + commands.first { $0.name == name || $0.aliases.contains(name) } + } + + static func parse(args: [String], command: Command) throws -> Invocation { + var inv = Invocation(command: command) + var i = 0 + var onlyPositionals = false + func option(long name: String) -> Opt? { + command.options.first { $0.long == name || $0.aliases.contains(name) } + } + while i < args.count { + let a = args[i] + i += 1 + if onlyPositionals || a == "-" || !a.hasPrefix("-") { inv.positionals.append(a); continue } + if a == "--" { onlyPositionals = true; continue } + var opt: Opt? + var inline: String? + var shown: String + if a.hasPrefix("--") { + let body = String(a.dropFirst(2)) + let parts = body.split(separator: "=", maxSplits: 1, omittingEmptySubsequences: false) + let name = String(parts[0]) + if parts.count == 2 { inline = String(parts[1]) } + if name == "help" { inv.helpRequested = true; continue } + guard let o = option(long: name) else { throw CLIError.usage("unknown flag: --\(name)") } + opt = o; shown = "--\(name)" + } else { + let chars = Array(a.dropFirst()) + let c = chars[0] + if c == "h" && chars.count == 1 { inv.helpRequested = true; continue } + guard let o = command.options.first(where: { $0.short == c }) else { + throw CLIError.usage("unknown shorthand flag: '\(c)' in -\(String(chars))") + } + if chars.count > 1 { + // -pVALUE and -p=VALUE + var rest = String(chars.dropFirst()) + if rest.hasPrefix("=") { rest.removeFirst() } + inline = rest + } + opt = o; shown = o.short.map { "-\($0), --\(o.long)" } ?? "--\(o.long)" + } + let o = opt! + let raw: String + if let kind = o.value { + if let inline { raw = inline } + else if let implicit = o.implicit { raw = implicit } + else if i < args.count { raw = args[i]; i += 1 } + else { throw CLIError.usage(a.hasPrefix("--") ? "flag needs an argument: \(shown)" : "flag needs an argument: '\(o.short.map(String.init) ?? "")' in -\(o.short.map(String.init) ?? "")") } + try validate(raw, kind: kind, flag: shown) + } else { + if let inline { + guard ["true", "false", "1", "0", "t", "f", "TRUE", "FALSE", "True", "False", "T", "F"].contains(inline) else { + throw CLIError.usage("invalid argument \"\(inline)\" for \"\(shown)\" flag: strconv.ParseBool: parsing \"\(inline)\": invalid syntax") + } + raw = ["true", "1", "t", "TRUE", "True", "T"].contains(inline) ? "true" : "false" + } else { raw = "true" } + } + if o.repeatable { inv.values[o.long, default: []].append(raw) } else { inv.values[o.long] = [raw] } + } + if !inv.helpRequested { try checkArgCount(inv.positionals.count, command: command) } + return inv + } + + static func validate(_ raw: String, kind: ValueKind, flag: String) throws { + func bad(_ why: String) -> CLIError { .usage("invalid argument \"\(raw)\" for \"\(flag)\" flag: \(why)") } + switch kind { + case .text, .model, .file: break + case .int(let r): + guard let n = Int(raw) else { throw bad("strconv.ParseInt: parsing \"\(raw)\": invalid syntax") } + guard r.contains(n) else { throw bad("value \(n) is outside \(r.lowerBound)...\(r.upperBound)") } + case .uint64: + guard UInt64(raw) != nil else { throw bad("strconv.ParseUint: parsing \"\(raw)\": invalid syntax") } + case .real(let r): + guard let d = Double(raw), d.isFinite else { throw bad("strconv.ParseFloat: parsing \"\(raw)\": invalid syntax") } + guard r.contains(d) else { throw bad("value \(raw) is outside \(r.lowerBound)...\(r.upperBound)") } + case .realAboveZero(let max): + guard let d = Double(raw), d.isFinite else { throw bad("strconv.ParseFloat: parsing \"\(raw)\": invalid syntax") } + guard d > 0 && d <= max else { throw bad("value \(raw) must be greater than 0 and at most \(max)") } + case .choice(let c): + guard c.contains(raw) else { throw bad("must be one of \(c.joined(separator: ", "))") } + case .choiceOrInt(let c): + guard c.contains(raw) || (Int(raw).map { $0 > 0 } ?? false) else { + throw bad("must be one of \(c.joined(separator: ", ")) or a positive number") + } + } + } + + static func checkArgCount(_ n: Int, command: Command) throws { + let required = command.args.filter { !$0.optional }.count + let variadic = command.args.contains { $0.variadic } + if variadic { + if n < required && command.exactArgWording { throw CLIError.usage("accepts \(required) arg(s), received \(n)") } + if n < required { throw CLIError.usage("requires at least \(required) arg(s), only received \(n)") } + } else if required == command.args.count { + if n != required { throw CLIError.usage("accepts \(required) arg(s), received \(n)") } + } else if n < required || n > command.args.count { + throw CLIError.usage("accepts between \(required) and \(command.args.count) arg(s), received \(n)") + } + } + + /// Commands whose name is close to a mistyped one (cobra: edit distance <= 2 or a prefix). + static func suggestions(for name: String, in commands: [Command]) -> [String] { + let lower = name.lowercased() + return commands.filter { !$0.hidden }.compactMap { c -> String? in + let hit = ([c.name] + c.aliases).contains { n in + editDistance(lower, n.lowercased()) <= 2 || n.lowercased().hasPrefix(lower) + } + return hit ? c.name : nil + } + } + + static func editDistance(_ a: String, _ b: String) -> Int { + let a = Array(a), b = Array(b) + var prev = Array(0...b.count) + for i in 1...max(a.count, 1) where !a.isEmpty { + var cur = [i] + [Int](repeating: 0, count: b.count) + for j in 1...max(b.count, 1) where !b.isEmpty { + cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (a[i - 1] == b[j - 1] ? 0 : 1)) + } + prev = cur + } + return a.isEmpty ? b.count : prev[b.count] + } +} diff --git a/Sources/MLXRCLI/PullProgress.swift b/Sources/MLXRCLI/PullProgress.swift new file mode 100644 index 0000000..80732c6 --- /dev/null +++ b/Sources/MLXRCLI/PullProgress.swift @@ -0,0 +1,76 @@ +import Foundation + +/// The progress line of `mlxr pull`, in Ollama's layout: +/// `"