diff --git a/docs/api/cbor.html b/docs/api/cbor.html index 35e6f39b2..f4debde10 100644 --- a/docs/api/cbor.html +++ b/docs/api/cbor.html @@ -23,8 +23,12 @@

Exports

  • CBOR_TAG
  • CBOR_SIMPLE
  • CBOR_FLOAT
    • +
  • CborDecodeOptions
    • +
  • default_decode_options
  • parse
    • +
  • parse_with
  • encode
    • +
  • encode_deterministic
  • type
  • is_null
  • is_undefined
    • @@ -196,11 +200,26 @@

    CBOR_SIMPLE const

    CBOR_FLOAT const

    CBOR_FLOAT = 7
    +
    +

    CborDecodeOptions struct

    + 
    struct CborDecodeOptions
    +

    What parse_with accepts. default_decode_options() is what parse uses: safe for input from an untrusted peer. The element and pair defaults are fxamacker/cbor's.

    +
    +
    +

    default_decode_options fn

    + 
    default_decode_options() -> CborDecodeOptions
    +

    parse fn

    parse(cbor_data) -> ptr!

    cbor_data: string

    -

    Parse one CBOR data item that is the whole of `cbor_data`. Returns (value_ptr, "") on success, (null, error) on failure. Caller owns the returned value and must free with cbor.free. Bytes left after the item are an error (#2470): `0102` used to parse as 1.

    +

    parse_with under default_decode_options(): duplicate map keys refused, 131072 elements or pairs per container, nesting to 256.

    +
    +
    +

    parse_with fn

    + 
    parse_with(cbor_data, opts) -> ptr!
    +

    cbor_data: string, opts: CborDecodeOptions

    +

    Parse one CBOR data item that is the whole of `cbor_data`, under `opts`. Returns (value_ptr, "") on success, (null, error) on failure. Caller owns the returned value and must free with cbor.free. Bytes left after the item are an error (#2470): `0102` used to parse as 1.

    encode fn

    @@ -208,6 +227,12 @@

    encode fn

    value: ptr

    Serialize a CBOR value tree to a binary string. Returns (encoded_bytes, "") on success, ("", error) on failure.

    +
    +

    encode_deterministic fn

    + 
    encode_deterministic(value) -> string!
    +

    value: ptr

    +

    Deterministic CBOR encoding (RFC 8949 §4.2 Core Deterministic Encoding).

    +

    type fn

    type(value) -> int
    diff --git a/docs/api/index.html b/docs/api/index.html index 464189b98..806197b1b 100644 --- a/docs/api/index.html +++ b/docs/api/index.html @@ -176,7 +176,7 @@

    casper

    cbor

    MIT License (https://opensource.org/licenses/MIT) Portions copyright (c) 2019-present Faye Amacker (https://github.com/fxamacker/cbor) Portions copyright (c) 2026 Aether Developers.

    - 46 exports + 50 exports

    clapae

    @@ -813,8 +813,12 @@

    Search Results

  • CBOR_TAGcbor
  • CBOR_SIMPLEcbor
  • CBOR_FLOATcbor
    • +
  • CborDecodeOptionscbor
    • +
  • default_decode_optionscbor
  • parsecbor
    • +
  • parse_withcbor
  • encodecbor
    • +
  • encode_deterministiccbor
  • typecbor
  • is_nullcbor
  • is_undefinedcbor
    • diff --git a/docs/stdlib-reference.md b/docs/stdlib-reference.md index 65b0685a2..408791b5b 100644 --- a/docs/stdlib-reference.md +++ b/docs/stdlib-reference.md @@ -23,7 +23,7 @@ header comment is the authoritative description. | `std.capsicum` | FreeBSD Capsicum capability-mode bindings. | 28 | [guide](../std/capsicum/README.md) · [source](../std/capsicum/module.ae) | | `std.cas` | Content-addressed store keyed by the sha256 of file contents. | 7 | [guide](../std/cas/README.md) · [source](../std/cas/module.ae) | | `std.casper` | FreeBSD Casper service delegation. | 8 | [guide](../std/casper/README.md) · [source](../std/casper/module.ae) | -| `std.cbor` | CBOR encoding and decoding (RFC 8949). | 46 | [guide](../std/cbor/README.md) · [source](../std/cbor/module.ae) | +| `std.cbor` | CBOR encoding and decoding (RFC 8949). | 50 | [guide](../std/cbor/README.md) · [source](../std/cbor/module.ae) | | `std.clapae` | Command-line argument parser, modelled on clap. | 32 | [guide](../std/clapae/README.md) · [source](../std/clapae/module.ae) | | `std.collections` | Dynamic list, hash map and packed int array, with the raw externs the alias modules re-export. | 44 | [guide](../std/collections/README.md) · [source](../std/collections/module.ae) | | `std.config` | Process-global immutable string to string store. | 6 | [guide](../std/config/README.md) · [source](../std/config/module.ae) | diff --git a/new_changelogs/20261029T182754Z-security-cbor-decode-limits.md b/new_changelogs/20261029T182754Z-security-cbor-decode-limits.md new file mode 100644 index 000000000..591e59528 --- /dev/null +++ b/new_changelogs/20261029T182754Z-security-cbor-decode-limits.md @@ -0,0 +1,13 @@ +- **`std.cbor` decodes untrusted input safely by default.** `parse` now + refuses duplicate map keys (`cbor: duplicate map key "grant"`), where two + keys are duplicates when they are the same value however they were encoded, + and limits each array or map to 131072 elements or pairs. It also refuses a + declared length or count that cannot fit in the bytes left before + allocating for it. `parse_with(data, opts)` takes a `CborDecodeOptions` + (`default_decode_options()` gives the defaults) to change the limits, allow + duplicates, or refuse indefinite-length items. `encode_deterministic` + writes RFC 8949 §4.2.1 core deterministic encoding: shortest forms, definite + lengths and sorted map keys. Duplicate detection and key sorting are + linear-logarithmic, so a map at the limit cannot be used to spend minutes + of CPU. Refusing duplicate keys is a change to `parse`'s behaviour; use + `parse_with` with `allow_duplicate_keys = 1` for the old one. diff --git a/std/cbor/README.md b/std/cbor/README.md index 812395d55..13aa1037b 100644 --- a/std/cbor/README.md +++ b/std/cbor/README.md @@ -59,11 +59,47 @@ and argument sizes. A value built with the constructors encodes in preferred serialization: the shortest head, and a float in the narrowest width that holds it exactly (`num(1.5)` is `f93e00`). +### Decode Options + +`parse_with(data, opts)` parses input with configurable decoder bounds and policy options: + +```aether,fragment +struct CborDecodeOptions { + max_depth: int // default 256 + max_array_elements: int // default 131072 + max_map_pairs: int // default 131072 + allow_duplicate_keys: int // default 0 (refuse) + allow_indefinite: int // default 1 +} +default_decode_options() -> CborDecodeOptions +parse_with(cbor_data: string, opts: CborDecodeOptions) -> ptr! +``` + +`parse(data)` calls `parse_with(data, default_decode_options())`. **By default, `parse` refuses duplicate map keys** (`cbor: duplicate map key ...`), preventing parser-differential vulnerabilities. + +- **Duplicate keys:** Two keys are duplicates when their decoded key values are equal as values (e.g. text key `"a"` encoded definite vs indefinite). Integer `1` and text `"1"` remain distinct keys. +- **Declared bounds & limits:** Before allocating or looping, declared string lengths, array element counts, and map pair counts are checked against remaining input bytes and option limits (`max_array_elements`, `max_map_pairs`). +- **Indefinite items:** Setting `allow_indefinite = 0` refuses indefinite-length strings, arrays, and maps (`cbor: indefinite-length items are disabled`). + +### Deterministic Encoding + +```aether,fragment +encode_deterministic(value: ptr) -> string! +``` + +`encode_deterministic` encodes values using RFC 8949 §4.2 Core Deterministic Encoding rules: +- Preferred serialization: shortest argument heads and narrowest exact float widths (half, single, double). +- Re-encodes indefinite-length strings, arrays, and maps as definite-length items. +- Map keys are sorted in the bytewise lexicographic order of their deterministic encodings. +- A map with duplicate keys returns an encoding error rather than silently encoding. +- Use `encode_deterministic` when CBOR messages must be hashed, signed, or compared byte-exact. `encode` retains its byte-for-byte roundtrip behavior. + `parse` treats its input as untrusted, and reads one data item that must be the whole input. It refuses, with an error rather than a value: - bytes after the item (`cbor: trailing bytes after the data item`): `0102` is not 1; +- duplicate map keys (`cbor: duplicate map key ...`); - anything that is not well-formed (RFC 8949 §3, every case of Appendix F): a reserved additional-information value, a break outside an indefinite-length item or where a map value belongs @@ -72,8 +108,7 @@ the whole input. It refuses, with an error rather than a value: bytes, a truncated head or body; - a text string, or a text chunk, that is not UTF-8 (`cbor: invalid UTF-8 in text string`); -- nesting deeper than 256 levels, the same limit as `std.json` - (`cbor: nesting deeper than 256`); +- nesting deeper than `max_depth` levels (default 256, the same limit as `std.json`); - a string length, or an array or map count, larger than the bytes left in the input, 2^63 and up included (`cbor: unexpected EOF`). @@ -96,6 +131,7 @@ specifies it. ## Exports -`parse`, `encode`, `diagnose`, `from_int`, `from_uint`, `from_nint`, `num`, +`parse`, `parse_with`, `CborDecodeOptions`, `default_decode_options`, `encode`, +`encode_deterministic`, `diagnose`, `from_int`, `from_uint`, `from_nint`, `num`, `str`, `arr`, `obj`, `boolean`, `null_value`, `get_long`, `get_int64`, `get_uint`, `get_nint`, `free`, and the `CBOR_*` type constants. diff --git a/std/cbor/module.ae b/std/cbor/module.ae index c12be2f97..553cda8c9 100644 --- a/std/cbor/module.ae +++ b/std/cbor/module.ae @@ -13,7 +13,7 @@ import std.encoding exports( CBOR_INT, CBOR_BYTES, CBOR_TEXT, CBOR_ARRAY, CBOR_MAP, CBOR_TAG, CBOR_SIMPLE, CBOR_FLOAT, - parse, encode, type, is_null, is_undefined, + CborDecodeOptions, default_decode_options, parse, parse_with, encode, encode_deterministic, type, is_null, is_undefined, get_bool, get_int, get_long, get_float, get_string, get_bytes, get_tag_val, get_tag_child, object_get, map_get, object_size, array_size, object_entry, obj, arr, str, bytes, num, from_int, boolean, null_value, undefined_value, tag, @@ -30,11 +30,32 @@ const CBOR_TAG = 5 const CBOR_SIMPLE = 6 const CBOR_FLOAT = 7 -// Deepest nesting parse accepts, the same bound as std.json. The decoder -// recurses once per level, so without a bound a few hundred kilobytes of -// one-element arrays overflow the stack. +// Deepest nesting parse accepts by default, the same bound as std.json. The +// decoder recurses once per level, so without a bound a few hundred kilobytes +// of one-element arrays overflow the stack. const CBOR_MAX_DEPTH = 256 +// What parse_with accepts. default_decode_options() is what parse uses: safe +// for input from an untrusted peer. The element and pair defaults are +// fxamacker/cbor's. +struct CborDecodeOptions { + max_depth: int + max_array_elements: int + max_map_pairs: int + allow_duplicate_keys: int + allow_indefinite: int +} + +default_decode_options() -> CborDecodeOptions { + CborDecodeOptions opts + opts.max_depth = 256 + opts.max_array_elements = 131072 + opts.max_map_pairs = 131072 + opts.allow_duplicate_keys = 0 + opts.allow_indefinite = 1 + return opts +} + // Memory model: a CborValue tree owns its text/byte-string payloads as an // AetherBytes buffer (str_bytes + str_len), not a `string` field. The choice // dates from before a heap.new struct released its string fields on @@ -75,6 +96,13 @@ struct CborValue { big: int // an integer outside the signed long range, see above } +struct MapPair { + key: ptr + val: ptr + enc_k: string + enc_v: string +} + // ---- str payload helpers (see Memory model above) ---- // Store `len` bytes of `src` (an AetherString) as cv's owned payload. @@ -142,6 +170,11 @@ struct Reader { buf: ptr // owned std.bytes copy of the input len: int offset: int + max_depth: int + max_array_elements: int + max_map_pairs: int + allow_duplicate_keys: int + allow_indefinite: int } // Read the byte at `off` (0..255). Bounds are checked by callers. @@ -372,6 +405,76 @@ fn rd_byte(b: ptr, off: int) -> int { return bytes.get(b, off) & 255 } +// Duplicate map keys (RFC 8949 §5.6): two keys are the same value exactly +// when their deterministic encodings match, whether either was written +// definite or indefinite. Once a map is parsed its key encodings are +// merge-sorted and neighbours compared. A pairwise scan was quadratic: +// 131072 small keys in about 400 KB cost billions of comparisons. (A hash +// set from std.map would do, but importing std.map collides with this +// module's own map_get.) Returns "" when every key is distinct, else the error. +fn duplicate_key_check(cv: *CborValue) -> string { + n = list.size(cv.map_keys) + if n < 2 { return "" } + pairs = list.new() + err = "" + i = 0 + while i < n { + k = list.get_raw(cv.map_keys, i) + ek, kerr = encode_deterministic(k) + if kerr != "" { + err = kerr + break + } + p = heap.new(MapPair) + p.key = k + p.val = null + p.enc_k = ek + p.enc_v = "" + _ = list.add(pairs, p as ptr) + i = i + 1 + } + m = list.size(pairs) + if err == "" { + sort_pairs(pairs, m) + j = 0 + while j < m - 1 { + pa = list.get_raw(pairs, j) as *MapPair + pb = list.get_raw(pairs, j + 1) as *MapPair + if bytes_lex_cmp(pa.enc_k, pb.enc_k) == 0 { + err = duplicate_key_error(pa.key) + break + } + j = j + 1 + } + } + j = 0 + while j < m { + p = list.get_raw(pairs, j) as *MapPair + string.release(p.enc_k) + heap.free(p) + j = j + 1 + } + list.free(pairs) + return err +} + +fn duplicate_key_error(key_cv: ptr) -> string { + if key_cv == null { return "cbor: duplicate map key null" } + k = key_cv as *CborValue + if k.type == CBOR_TEXT { + s = str_payload_string(k) + esc = escape_string(s) + string.release(s) + msg = string.concat("cbor: duplicate map key ", esc) + string.release(esc) + return msg + } + k_diag = to_diagnose_string(key_cv) + msg = string.concat("cbor: duplicate map key ", k_diag) + string.release(k_diag) + return msg +} + // An indefinite-length string (RFC 8949 §3.2.3): definite-length strings of // the same major type up to a break. A chunk of another type, or a nested // indefinite-length string (`5f5f4101ffff`), is not well-formed; it used to @@ -379,6 +482,9 @@ fn rd_byte(b: ptr, off: int) -> int { // diagnoses as it came. fn parse_indefinite_string(r: ptr, mt: int, depth: int) -> (ptr, string) { reader = r as *Reader + if reader.allow_indefinite == 0 { + return null, "cbor: indefinite-length items are disabled" + } cv = heap.new(CborValue) cv.type = CBOR_BYTES if mt == 3 { @@ -457,8 +563,8 @@ fn parse_float(reader: *Reader, width: int) -> (ptr, string) { // an error, as is invalid UTF-8 in a text string (§3.1). parse_item(r: ptr, depth: int) -> (ptr, string) { reader = r as *Reader - if depth > CBOR_MAX_DEPTH { - return null, "cbor: nesting deeper than 256" + if depth > reader.max_depth { + return null, string.concat("cbor: nesting deeper than ", string.from_int(reader.max_depth)) } if reader.offset >= reader.len { return null, "cbor: unexpected EOF" @@ -507,7 +613,8 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { string.release(err) // Compared in long against what is left: narrowing a length of 2^31 // or more to int first made it negative and passed it as empty. - if len_val < 0 || len_val > ((reader.len - reader.offset) as long) { return null, "cbor: unexpected EOF" } + rem = (reader.len - reader.offset) as long + if len_val < 0 || len_val > rem { return null, "cbor: unexpected EOF" } if mt == 3 && !valid_utf8(reader.buf, reader.offset, len_val as int) { return null, "cbor: invalid UTF-8 in text string" } @@ -527,7 +634,12 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { cv.type = CBOR_ARRAY cv.list_val = list.new() if ai == 31 { + if reader.allow_indefinite == 0 { + free_value(cv) + return null, "cbor: indefinite-length items are disabled" + } cv.indefinite = 1 + arr_count = 0 while true { if reader.offset >= reader.len { free_value(cv) @@ -538,6 +650,10 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { reader.offset = reader.offset + 1 break } + if arr_count >= reader.max_array_elements { + free_value(cv) + return null, "cbor: max array elements exceeded" + } child, err = parse_item(r, depth + 1) if err != "" { free_value(cv) @@ -545,6 +661,7 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { } string.release(err) _ = list.add(cv.list_val, child) + arr_count = arr_count + 1 } return cv as ptr, "" } @@ -558,12 +675,18 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { // Every element takes at least one byte, so a count beyond what is // left is truncated input, refused before the loop rather than // narrowed to int (2^32 became 0 elements). - if len_val < 0 || len_val > ((reader.len - reader.offset) as long) { + rem = (reader.len - reader.offset) as long + if len_val < 0 || len_val > rem { free_value(cv) return null, "cbor: unexpected EOF" } + if len_val > (reader.max_array_elements as long) { + free_value(cv) + return null, "cbor: max array elements exceeded" + } i = 0 - while i < (len_val as int) { + n = len_val as int + while i < n { child, err = parse_item(r, depth + 1) if err != "" { free_value(cv) @@ -583,7 +706,12 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { cv.map_keys = list.new() cv.map_values = list.new() if ai == 31 { + if reader.allow_indefinite == 0 { + free_value(cv) + return null, "cbor: indefinite-length items are disabled" + } cv.indefinite = 1 + map_count = 0 while true { if reader.offset >= reader.len { free_value(cv) @@ -594,6 +722,10 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { reader.offset = reader.offset + 1 break } + if map_count >= reader.max_map_pairs { + free_value(cv) + return null, "cbor: max map pairs exceeded" + } key_cv, err = parse_item(r, depth + 1) if err != "" { free_value(cv) @@ -611,6 +743,14 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { string.release(err2) _ = list.add(cv.map_keys, key_cv) _ = list.add(cv.map_values, val_cv) + map_count = map_count + 1 + } + if reader.allow_duplicate_keys == 0 { + dup_err = duplicate_key_check(cv as *CborValue) + if dup_err != "" { + free_value(cv) + return null, dup_err + } } return cv as ptr, "" } @@ -622,12 +762,18 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { string.release(err) cv.ind = ind_of(ai) // Every entry is a key and a value, at least two bytes. - if len_val < 0 || len_val > ((reader.len - reader.offset) as long) / 2 { + rem = (reader.len - reader.offset) as long + if len_val < 0 || len_val > rem / 2 { free_value(cv) return null, "cbor: unexpected EOF" } + if len_val > (reader.max_map_pairs as long) { + free_value(cv) + return null, "cbor: max map pairs exceeded" + } i = 0 - while i < (len_val as int) { + n = len_val as int + while i < n { key_cv, err = parse_item(r, depth + 1) if err != "" { free_value(cv) @@ -645,6 +791,13 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { _ = list.add(cv.map_values, val_cv) i = i + 1 } + if reader.allow_duplicate_keys == 0 { + dup_err = duplicate_key_check(cv as *CborValue) + if dup_err != "" { + free_value(cv) + return null, dup_err + } + } return cv as ptr, "" } if mt == 6 { @@ -700,11 +853,11 @@ parse_item(r: ptr, depth: int) -> (ptr, string) { return null, "cbor: invalid additional information" } -// Parse one CBOR data item that is the whole of `cbor_data`. Returns -// (value_ptr, "") on success, (null, error) on failure. Caller owns the -// returned value and must free with cbor.free. Bytes left after the item are -// an error (#2470): `0102` used to parse as 1. -parse(cbor_data: string) -> ptr! { +// Parse one CBOR data item that is the whole of `cbor_data`, under `opts`. +// Returns (value_ptr, "") on success, (null, error) on failure. Caller owns +// the returned value and must free with cbor.free. Bytes left after the item +// are an error (#2470): `0102` used to parse as 1. +parse_with(cbor_data: string, opts: CborDecodeOptions) -> ptr! { ok = "" dlen = string.length(cbor_data) // Copy the input into an owned bytes buffer. We must NOT store the @@ -718,6 +871,12 @@ parse(cbor_data: string) -> ptr! { r.buf = buf r.len = dlen r.offset = 0 + r.max_depth = opts.max_depth + r.max_array_elements = opts.max_array_elements + r.max_map_pairs = opts.max_map_pairs + r.allow_duplicate_keys = opts.allow_duplicate_keys + r.allow_indefinite = opts.allow_indefinite + res, err = parse_item(r as ptr, 0) trailing = r.offset < r.len bytes.free(r.buf) @@ -735,6 +894,12 @@ parse(cbor_data: string) -> ptr! { return res, ok } +// parse_with under default_decode_options(): duplicate map keys refused, +// 131072 elements or pairs per container, nesting to 256. +parse(cbor_data: string) -> ptr! { + return parse_with(cbor_data, default_decode_options()) +} + // ---- ENCODER IMPLEMENTATION ---- write_byte(w: ptr, b: int) { @@ -804,6 +969,10 @@ encode_head(w: ptr, major: int, val: long, ind: int) { } } +fn encode_head_preferred(w: ptr, major: int, val: long) { + encode_head(w, major, val, 0) +} + // ---- Float widths: ind 2 is half, 3 single, 4 double ---- // Whether a nonzero finite value whose leading bit is 2^e and lowest set bit @@ -1060,6 +1229,291 @@ stringify(value: ptr) -> string! { return encode(value) } +fn bytes_lex_cmp(s1: string, s2: string) -> int { + l1 = string.length(s1) + l2 = string.length(s2) + min_l = l1 + if l2 < min_l { min_l = l2 } + i = 0 + while i < min_l { + b1 = string.char_at_n(s1, l1, i) & 255 + b2 = string.char_at_n(s2, l2, i) & 255 + if b1 < b2 { return -1 } + if b1 > b2 { return 1 } + i = i + 1 + } + if l1 < l2 { return -1 } + if l1 > l2 { return 1 } + return 0 +} + +// Sort *MapPair entries by the bytes of their encoded keys (RFC 8949 +// §4.2.1). A merge sort: an insertion sort was quadratic in the pair count. +// Encodings are prefix-free, so bytewise order needs no tie-break. +fn sort_pairs(pairs: ptr, n: int) { + tmp = list.new() + i = 0 + while i < n { + _ = list.add(tmp, list.get_raw(pairs, i)) + i = i + 1 + } + src = pairs + dst = tmp + width = 1 + while width < n { + lo = 0 + while lo < n { + mid = lo + width + if mid > n { mid = n } + hi = lo + 2 * width + if hi > n { hi = n } + a = lo + b = mid + k = lo + while k < hi { + take_a = 0 + if a < mid { + if b >= hi { + take_a = 1 + } else { + pa = list.get_raw(src, a) as *MapPair + pb = list.get_raw(src, b) as *MapPair + if bytes_lex_cmp(pa.enc_k, pb.enc_k) <= 0 { take_a = 1 } + } + } + if take_a == 1 { + list.set(dst, k, list.get_raw(src, a)) + a = a + 1 + } else { + list.set(dst, k, list.get_raw(src, b)) + b = b + 1 + } + k = k + 1 + } + lo = hi + } + t = src + src = dst + dst = t + width = width * 2 + } + if src != pairs { + i = 0 + while i < n { + list.set(pairs, i, list.get_raw(src, i)) + i = i + 1 + } + } + list.free(tmp) +} + +encode_deterministic_item(w: ptr, cv: ptr) -> string { + if cv == null { + write_byte(w, 246) + return "" + } + + cv_val = cv as *CborValue + if cv_val.type == CBOR_INT { + v = cv_val.int_val + if cv_val.big == 1 { + encode_head_preferred(w, 0, v) + } else if cv_val.big == 2 { + encode_head_preferred(w, 1, v) + } else if v >= 0 { + encode_head_preferred(w, 0, v) + } else { + encode_head_preferred(w, 1, -1 - v) + } + return "" + } + if cv_val.type == CBOR_BYTES || cv_val.type == CBOR_TEXT { + major = 2 + if cv_val.type == CBOR_TEXT { + major = 3 + } + // RFC 8949 §4.2.1: Re-encode indefinite string as definite string using payload + len = cv_val.str_len + encode_head_preferred(w, major, len as long) + if len > 0 { + s = str_payload_string(cv_val) + write_string(w, s, len) + string.release(s) + } + return "" + } + if cv_val.type == CBOR_ARRAY { + n = list.size(cv_val.list_val) + encode_head_preferred(w, 4, n as long) + i = 0 + while i < n { + item, _ = list.get(cv_val.list_val, i) + err = encode_deterministic_item(w, item) + if err != "" { return err } + i = i + 1 + } + return "" + } + if cv_val.type == CBOR_MAP { + n = list.size(cv_val.map_keys) + if n == 0 { + encode_head_preferred(w, 5, 0 as long) + return "" + } + pairs = list.new() + i = 0 + var map_err = "" + while i < n { + k, _ = list.get(cv_val.map_keys, i) + v, _ = list.get(cv_val.map_values, i) + ek, kerr = encode_deterministic(k) + if kerr != "" { + map_err = kerr + break + } + ev, verr = encode_deterministic(v) + if verr != "" { + string.release(ek) + map_err = verr + break + } + p = heap.new(MapPair) + p.key = k + p.val = v + p.enc_k = ek + p.enc_v = ev + _ = list.add(pairs, p as ptr) + i = i + 1 + } + if map_err != "" { + num_pairs = list.size(pairs) + p_idx = 0 + while p_idx < num_pairs { + p = list.get_raw(pairs, p_idx) as *MapPair + string.release(p.enc_k) + string.release(p.enc_v) + heap.free(p) + p_idx = p_idx + 1 + } + list.free(pairs) + return map_err + } + + sort_pairs(pairs, n) + + // Check for duplicate key encodings + var check_i = 0 + while check_i < n - 1 { + p_a = list.get_raw(pairs, check_i) as *MapPair + p_b = list.get_raw(pairs, check_i + 1) as *MapPair + if bytes_lex_cmp(p_a.enc_k, p_b.enc_k) == 0 { + map_err = duplicate_key_error(p_a.key) + break + } + check_i = check_i + 1 + } + + if map_err != "" { + num_pairs = list.size(pairs) + p_idx = 0 + while p_idx < num_pairs { + p = list.get_raw(pairs, p_idx) as *MapPair + string.release(p.enc_k) + string.release(p.enc_v) + heap.free(p) + p_idx = p_idx + 1 + } + list.free(pairs) + return map_err + } + + // Write map head and sorted pairs + encode_head_preferred(w, 5, n as long) + var write_i = 0 + while write_i < n { + p = list.get_raw(pairs, write_i) as *MapPair + write_string(w, p.enc_k, string.length(p.enc_k)) + write_string(w, p.enc_v, string.length(p.enc_v)) + write_i = write_i + 1 + } + + // Clean up temporary pairs list and string references + p_idx = 0 + while p_idx < n { + p = list.get_raw(pairs, p_idx) as *MapPair + string.release(p.enc_k) + string.release(p.enc_v) + heap.free(p) + p_idx = p_idx + 1 + } + list.free(pairs) + return "" + } + if cv_val.type == CBOR_TAG { + encode_head_preferred(w, 6, cv_val.tag_val) + err = encode_deterministic_item(w, cv_val.tag_child) + return err + } + if cv_val.type == CBOR_SIMPLE { + v = cv_val.int_val + if v < 24 { + write_byte(w, 224 | (v as int)) + } else { + write_byte(w, 224 | 24) + write_byte(w, v as int) + } + return "" + } + if cv_val.type == CBOR_FLOAT { + var width = 4 + if cv_val.ind != 0 && is_nan(cv_val.float_val) { + width = float_preferred(cv_val) + } else { + width = float_width_for(cv_val.float_val) + } + if width == 2 { + write_byte(w, 249) + if cv_val.ind == 2 && is_nan(cv_val.float_val) { + write_be16(w, cv_val.int_val as int) + } else { + write_be16(w, half_bits(cv_val.float_val)) + } + } else if width == 3 { + write_byte(w, 250) + if cv_val.ind == 3 && is_nan(cv_val.float_val) { + write_be32(w, cv_val.int_val as int) + } else { + write_be32(w, single_bits(cv_val.float_val) as int) + } + } else { + write_byte(w, 251) + if cv_val.ind == 4 && is_nan(cv_val.float_val) { + write_be64(w, cv_val.int_val) + } else { + write_be64(w, mem.bits_of_float(cv_val.float_val)) + } + } + return "" + } + return "cbor: unsupported CBOR value type for encoding" +} + +// Deterministic CBOR encoding (RFC 8949 §4.2 Core Deterministic Encoding). +encode_deterministic(value: ptr) -> string! { + w = heap.new(Writer) + w.buf = bytes.new(32) + w.offset = 0 + err = encode_deterministic_item(w as ptr, value) + if err != "" { + bytes.free(w.buf) + heap.free(w) + return "", err + } + res = bytes.finish(w.buf, w.offset) + heap.free(w) + return res, "" +} + // ---- ERGONOMIC GETTERS ---- type(value: ptr) -> int { diff --git a/std/cbor/test_cbor_hardening.ae b/std/cbor/test_cbor_hardening.ae new file mode 100644 index 000000000..5e2febba4 --- /dev/null +++ b/std/cbor/test_cbor_hardening.ae @@ -0,0 +1,374 @@ +// Hardening test suite for std.cbor +// Tests CborDecodeOptions, duplicate map keys, length bombs, indefinite length limits, +// and RFC 8949 §4.2 Core Deterministic Encoding. +// +// Reference semantics and test vectors cited from fxamacker/cbor (MIT). + +import std.cbor +import std.encoding +import std.string +import std.bytes +import std.mem + +extern exit(code: int) + +fn expect_parse_with_error(hex_data: string, opts: cbor.CborDecodeOptions, expected_err: string) -> int { + binary_data, err = encoding.hex_decode(hex_data) + if err != "" { + println(" FAIL: hex decode failed for ${hex_data}: ${err}") + return 1 + } + cv_ptr, err2 = cbor.parse_with(binary_data, opts) + if err2 != expected_err { + diag, _ = cbor.diagnose(cv_ptr) + println(" FAIL parse_with: ${hex_data} should be refused") + println(" got: err='${err2}' value=${diag}") + println(" expected: err='${expected_err}'") + cbor.free(cv_ptr) + return 1 + } + if cv_ptr != null { + cbor.free(cv_ptr) + } + return 0 +} + +fn expect_parse_error(hex_data: string, expected_err: string) -> int { + return expect_parse_with_error(hex_data, cbor.default_decode_options(), expected_err) +} + +fn run_parse_with_ok(hex_data: string, opts: cbor.CborDecodeOptions, expected_diag: string) -> int { + binary_data, err = encoding.hex_decode(hex_data) + if err != "" { + println(" FAIL: hex decode failed for ${hex_data}: ${err}") + return 1 + } + cv_ptr, err2 = cbor.parse_with(binary_data, opts) + if err2 != "" { + println(" FAIL parse_with ok: ${hex_data} failed with err: ${err2}") + return 1 + } + diag, _ = cbor.diagnose(cv_ptr) + if diag != expected_diag { + println(" FAIL parse_with diag mismatch for ${hex_data}: got '${diag}', want '${expected_diag}'") + cbor.free(cv_ptr) + return 1 + } + cbor.free(cv_ptr) + return 0 +} + +// Citation: fxamacker/cbor decode_test.go TestDecodeDupMapKey +fn test_duplicate_keys() -> int { + f = 0 + // Same text key "grant" ("grant": "none", "grant": "all") -> refused by default + // hex: a2 65 67 72 61 6e 74 64 6e 6f 6e 65 65 67 72 61 6e 74 63 61 6c 6c + dup_text = "a2656772616e74646e6f6e65656772616e7463616c6c" + f = f + expect_parse_error(dup_text, "cbor: duplicate map key \"grant\"") + + // With allow_duplicate_keys = 1, duplicate text key is accepted + opts_allow_dup = cbor.default_decode_options() + opts_allow_dup.allow_duplicate_keys = 1 + f = f + run_parse_with_ok(dup_text, opts_allow_dup, "{\"grant\": \"none\", \"grant\": \"all\"}") + + // Duplicate text key where one key is definite "a" (6161) and one is indefinite "a" (7f6161ff) + // hex: a2 61 61 01 7f 61 61 ff 02 + dup_indef_key = "a26161017f6161ff02" + f = f + expect_parse_error(dup_indef_key, "cbor: duplicate map key \"a\"") + + // Integer keys duplicate: {1: 10, 1: 20} + // hex: a2 01 0a 01 14 + dup_int = "a2010a0114" + f = f + expect_parse_error(dup_int, "cbor: duplicate map key 1") + + // Nested map duplicate keys: {1: {"grant": "none", "grant": "all"}} + // hex: a1 01 a2 65 67 72 61 6e 74 64 6e 6f 6e 65 65 67 72 61 6e 74 63 61 6c 6c + nested_dup = "a101a2656772616e74646e6f6e65656772616e7463616c6c" + f = f + expect_parse_error(nested_dup, "cbor: duplicate map key \"grant\"") + + // Integer key 1 and text key "1" in same map: {1: "int", "1": "str"} -> accepted (different types) + // hex: a2 01 63 69 6e 74 61 31 63 73 74 72 + int_vs_str_key = "a20163696e74613163737472" + f = f + run_parse_with_ok(int_vs_str_key, cbor.default_decode_options(), "{1: \"int\", \"1\": \"str\"}") + + if f == 0 { println(" PASS: duplicate map key tests") } + return f +} + +// Citation: fxamacker/cbor decode_test.go TestDecodeLengthBomb +fn test_length_bombs() -> int { + f = 0 + // Byte string claiming ~2^32 bytes in 5 bytes input + f = f + expect_parse_error("5affffffff", "cbor: unexpected EOF") + f = f + expect_parse_error("5a80000000", "cbor: unexpected EOF") + + // Byte string claiming ~2^64 bytes in 9 bytes input + f = f + expect_parse_error("5bffffffffffffffff", "cbor: unexpected EOF") + + // Array claiming ~2^32 elements in 5 bytes input + f = f + expect_parse_error("9affffffff", "cbor: unexpected EOF") + f = f + expect_parse_error("9a80000000", "cbor: unexpected EOF") + + // Array claiming ~2^64 elements in 9 bytes input + f = f + expect_parse_error("9bffffffffffffffff", "cbor: unexpected EOF") + + // Map claiming ~2^32 pairs in 5 bytes input + f = f + expect_parse_error("baffffffff", "cbor: unexpected EOF") + f = f + expect_parse_error("ba80000000", "cbor: unexpected EOF") + + // Map claiming ~2^64 pairs in 9 bytes input + f = f + expect_parse_error("bbffffffffffffffff", "cbor: unexpected EOF") + + if f == 0 { println(" PASS: length bomb tests") } + return f +} + +// Citation: fxamacker/cbor decode_opts_test.go TestDecOptions +fn test_option_boundaries() -> int { + f = 0 + + // --- max_depth --- + opts_depth = cbor.default_decode_options() + opts_depth.max_depth = 4 + // Nesting depth 4: [[[[1]]]] -> 4 nesting levels around integer 1 -> accepted + // hex: 81 81 81 81 01 + nest4 = "8181818101" + f = f + run_parse_with_ok(nest4, opts_depth, "[[[[1]]]]") + + // Nesting depth 5: [[[[[1]]]]] -> 5 nesting levels -> refused with custom limit message + // hex: 81 81 81 81 81 01 + nest5 = "818181818101" + f = f + expect_parse_with_error(nest5, opts_depth, "cbor: nesting deeper than 4") + + // --- max_array_elements --- + opts_arr = cbor.default_decode_options() + opts_arr.max_array_elements = 3 + // Array of 3 elements [1, 2, 3] -> accepted + // hex: 83 01 02 03 + arr3 = "83010203" + f = f + run_parse_with_ok(arr3, opts_arr, "[1, 2, 3]") + + // Array of 4 elements [1, 2, 3, 4] -> refused + // hex: 84 01 02 03 04 + arr4 = "8401020304" + f = f + expect_parse_with_error(arr4, opts_arr, "cbor: max array elements exceeded") + + // Indefinite array with 3 elements -> accepted + // hex: 9f 01 02 03 ff + indef_arr3 = "9f010203ff" + f = f + run_parse_with_ok(indef_arr3, opts_arr, "[_ 1, 2, 3]") + + // Indefinite array with 4 elements -> refused + // hex: 9f 01 02 03 04 ff + indef_arr4 = "9f01020304ff" + f = f + expect_parse_with_error(indef_arr4, opts_arr, "cbor: max array elements exceeded") + + // --- max_map_pairs --- + opts_map = cbor.default_decode_options() + opts_map.max_map_pairs = 2 + // Map with 2 pairs {1: 10, 2: 20} -> accepted + // hex: a2 01 0a 02 14 + map2 = "a2010a0214" + f = f + run_parse_with_ok(map2, opts_map, "{1: 10, 2: 20}") + + // Map with 3 pairs {1: 10, 2: 20, 3: 30} -> refused + // hex: a3 01 0a 02 14 03 1e + map3 = "a3010a0214031e" + f = f + expect_parse_with_error(map3, opts_map, "cbor: max map pairs exceeded") + + // Indefinite map with 2 pairs -> accepted + // hex: bf 01 0a 02 14 ff + indef_map2 = "bf010a0214ff" + f = f + run_parse_with_ok(indef_map2, opts_map, "{_ 1: 10, 2: 20}") + + // Indefinite map with 3 pairs -> refused + // hex: bf 01 0a 02 14 03 1e ff + indef_map3 = "bf010a0214031eff" + f = f + expect_parse_with_error(indef_map3, opts_map, "cbor: max map pairs exceeded") + + // --- allow_indefinite --- + opts_no_indef = cbor.default_decode_options() + opts_no_indef.allow_indefinite = 0 + // Indefinite string, array, and map refused + f = f + expect_parse_with_error("7f6161ff", opts_no_indef, "cbor: indefinite-length items are disabled") + f = f + expect_parse_with_error("9f0102ff", opts_no_indef, "cbor: indefinite-length items are disabled") + f = f + expect_parse_with_error("bf616101ff", opts_no_indef, "cbor: indefinite-length items are disabled") + + if f == 0 { println(" PASS: option boundary tests") } + return f +} + +// Citation: fxamacker/cbor encode_test.go TestCoreDetEnc +fn test_deterministic_encoding() -> int { + f = 0 + + // Unsorted map keys: {"b": 2, "a": 1} + // Parses and re-encodes deterministically with keys sorted: {"a": 1, "b": 2} + // hex: a2 61 62 02 61 61 01 -> sorted hex: a2 61 61 01 61 62 02 + unsorted_map_hex = "a2616202616101" + bin1, _ = encoding.hex_decode(unsorted_map_hex) + v1, err1 = cbor.parse(bin1) + if err1 != "" { println(" FAIL det enc parse unsorted: ${err1}") return 1 } + det1, derr1 = cbor.encode_deterministic(v1) + if derr1 != "" { println(" FAIL det enc unsorted: ${derr1}") cbor.free(v1); return 1 } + det1_hex = encoding.hex_encode(string.bytes(det1)[0..string.length(det1)]) + if det1_hex != "a2616101616202" { + println(" FAIL det enc map key sort: got ${det1_hex}, want a2616101616202") + f = f + 1 + } + + // Roundtrip property: parse(encode_deterministic(v)) then encode_deterministic again gives identical bytes + v1_back, _ = cbor.parse(det1) + det1_again, _ = cbor.encode_deterministic(v1_back) + det1_again_hex = encoding.hex_encode(string.bytes(det1_again)[0..string.length(det1_again)]) + if det1_again_hex != det1_hex { + println(" FAIL det enc roundtrip mismatch: got ${det1_again_hex}, want ${det1_hex}") + f = f + 1 + } + cbor.free(v1) + cbor.free(v1_back) + string.release(det1) + string.release(det1_again) + + // Non-preferred integer serialization: 255 encoded in 2-byte head (0x1900ff) + // encode_deterministic re-encodes to preferred 1-byte head (0x18ff) + bin2, _ = encoding.hex_decode("1900ff") + v2, _ = cbor.parse(bin2) + det2, _ = cbor.encode_deterministic(v2) + det2_hex = encoding.hex_encode(string.bytes(det2)[0..string.length(det2)]) + if det2_hex != "18ff" { + println(" FAIL det enc preferred int: got ${det2_hex}, want 18ff") + f = f + 1 + } + cbor.free(v2) + string.release(det2) + + // Float width narrowing: 1.5 in double precision (fb3ff8000000000000) + // encode_deterministic re-encodes to half precision (f93e00) + bin3, _ = encoding.hex_decode("fb3ff8000000000000") + v3, _ = cbor.parse(bin3) + det3, _ = cbor.encode_deterministic(v3) + det3_hex = encoding.hex_encode(string.bytes(det3)[0..string.length(det3)]) + if det3_hex != "f93e00" { + println(" FAIL det enc float 1.5 shrink to half: got ${det3_hex}, want f93e00") + f = f + 1 + } + cbor.free(v3) + string.release(det3) + + // Float 100000.0 (single float, fa47c35000) stays single float + bin4, _ = encoding.hex_decode("fa47c35000") + v4, _ = cbor.parse(bin4) + det4, _ = cbor.encode_deterministic(v4) + det4_hex = encoding.hex_encode(string.bytes(det4)[0..string.length(det4)]) + if det4_hex != "fa47c35000" { + println(" FAIL det enc float 100000.0 single: got ${det4_hex}, want fa47c35000") + f = f + 1 + } + cbor.free(v4) + string.release(det4) + + // Indefinite array re-encoded as definite array: [_ 1, 2] (9f0102ff) -> [1, 2] (820102) + bin5, _ = encoding.hex_decode("9f0102ff") + v5, _ = cbor.parse(bin5) + det5, _ = cbor.encode_deterministic(v5) + det5_hex = encoding.hex_encode(string.bytes(det5)[0..string.length(det5)]) + if det5_hex != "820102" { + println(" FAIL det enc indef array to def: got ${det5_hex}, want 820102") + f = f + 1 + } + cbor.free(v5) + string.release(det5) + + // Indefinite string re-encoded as definite string: (_ "a", "b") (7f61616162ff) -> "ab" (626162) + bin6, _ = encoding.hex_decode("7f61616162ff") + v6, _ = cbor.parse(bin6) + det6, _ = cbor.encode_deterministic(v6) + det6_hex = encoding.hex_encode(string.bytes(det6)[0..string.length(det6)]) + if det6_hex != "626162" { + println(" FAIL det enc indef string to def: got ${det6_hex}, want 626162") + f = f + 1 + } + cbor.free(v6) + string.release(det6) + + // Map with duplicate keys passed to encode_deterministic returns an error + m_dup = cbor.obj() + _ = cbor.map_push(m_dup, cbor.str("grant"), cbor.str("none")) + _ = cbor.map_push(m_dup, cbor.str("grant"), cbor.str("all")) + _, det_dup_err = cbor.encode_deterministic(m_dup) + if det_dup_err != "cbor: duplicate map key \"grant\"" { + println(" FAIL det enc map with duplicate key: got err '${det_dup_err}', want 'cbor: duplicate map key \"grant\"'") + f = f + 1 + } + cbor.free(m_dup) + + if f == 0 { println(" PASS: deterministic encoding tests") } + return f +} + +// Duplicate detection and key sorting at the default pair limit. A pairwise +// duplicate scan or an insertion sort is quadratic here: 131072 keys made +// billions of comparisons, so either regression shows as this test hanging. +fn test_large_maps() -> int { + f = 0 + n = 131072 + m = cbor.obj() + i = 0 + while i < n { + // descending, so a sort has work to do + _ = cbor.map_push(m, cbor.from_int((n - i) as long), cbor.from_int(0 as long)) + i = i + 1 + } + enc, eerr = cbor.encode(m) + cbor.free(m) + if eerr != "" { + println(" FAIL large map encode: ${eerr}") + return 1 + } + back, perr = cbor.parse(enc) + if perr != "" { + println(" FAIL large map with ${n} distinct keys refused: ${perr}") + return 1 + } + det, derr = cbor.encode_deterministic(back) + cbor.free(back) + if derr != "" { + println(" FAIL large map deterministic encode: ${derr}") + return 1 + } + sorted, serr = cbor.parse(det) + if serr != "" { + println(" FAIL large map re-parse: ${serr}") + return 1 + } + k0, _v0, _e0 = cbor.object_entry(sorted, 0) + kl, _vl, _el = cbor.object_entry(sorted, n - 1) + first = cbor.get_long(k0) + last = cbor.get_long(kl) + if first != 1 || last != (n as long) { + println(" FAIL large map keys not sorted: first ${first}, last ${last}") + f = f + 1 + } + cbor.free(sorted) + if f == 0 { println(" PASS: ${n}-key map parses and sorts") } + return f +} + +main() { + println("=== test_cbor_hardening ===") + fails = 0 + + fails = fails + test_duplicate_keys() + fails = fails + test_length_bombs() + fails = fails + test_option_boundaries() + fails = fails + test_deterministic_encoding() + fails = fails + test_large_maps() + + if fails != 0 { + println("CBOR HARDENING: ${fails} failure(s)") + exit(1) + } + println("PASS: test_cbor_hardening") + println("=== test_cbor_hardening passed ===") +}