From fc39a183c3be9692057e00c3986e07e84706994a Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Sun, 27 Sep 2026 15:37:09 +0200 Subject: [PATCH] Decide sooner that a region is too slow to scan On a drive covering 34 KiB/s, the rate rule needed eight chunks and a minute of I/O, and each chunk could spend 30 s drilling: four minutes before anything could be decided. - A drill-down is cut at 10 s of I/O, about three times what drilling a healthy 1 MiB chunk costs (256 reads at a 13.6 ms sector median). - The rate rule judges after 3 chunks and 20 s of I/O; a healthy drive never spends 20 s on three chunks. - A skip that starts right where a cut chunk ended extends that region rather than starting a second one. Co-Authored-By: Claude Opus 5.5 (1M context) --- DESIGN.md | 14 ++++++++----- docs/guide.html | 2 +- hddscan.8 | 4 ++-- hddscan.c | 53 +++++++++++++++++++++++++++++++++++-------------- tests/cli.sh | 4 ++-- 5 files changed, 52 insertions(+), 25 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 08392c2..271c8bd 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -333,8 +333,10 @@ to scan when, over a trailing window of chunks, either: - three quarters of the last 64 MiB were over budget, or - scanning them ran below the `--min-rate` floor, the same floor that calls a whole drive too slow to test. That is measured on I/O time, with every - drill-down, retry and cache-busting seek counted, once there are at least 8 - chunks and a minute of I/O to judge. + drill-down, retry and cache-busting seek counted, once there are at least 3 + chunks and 20 s of I/O to judge. A healthy drive never spends 20 s on three + chunks; that is a twentieth of the floor. Looser than this, a drive covering + 34 KiB/s spent four minutes before anything could be decided. Then it stops reading. It probes 64 MiB 1 GiB further on, and again, until a probe comes back fast, then bisects between the last slow probe and that one @@ -354,9 +356,11 @@ as over. The steps are fixed rather than doubling, so a healthy island more than a gigabyte wide is still found. And whatever the window says, one chunk may not eat the scan. A drill-down -that has spent 30 s of I/O on one chunk stops, and the whole chunk is -condemned: the sectors drilled so far stay in the findings, and the rest is -recorded as unread. A run of such chunks becomes one region. +that has spent 10 s of I/O on one chunk, about three times what drilling a +healthy 1 MiB chunk costs, stops, and the whole chunk is condemned. The +sectors drilled so far stay in the findings, and the rest is recorded as +unread. A run of such chunks becomes one region, and so does a cut chunk +followed by a skipped region. I/O time rather than wall-clock time is what gets judged, because it counts only what the drive did. It is also what `HDDSCAN_SLOW` can make exact for the diff --git a/docs/guide.html b/docs/guide.html index 4ec0ade..4b27ab2 100644 --- a/docs/guide.html +++ b/docs/guide.html @@ -507,7 +507,7 @@

Test mode

--second-pass
after --mode write, re-read the whole device and re-verify the pattern (catches slow data decay)
--skip-slow
-
when a region is too slow to scan -- most chunks over the last 64 MiB missed their budget, or the scan there, drill-downs and retries included, ran under the --min-rate floor -- stop reading: probe 1 GiB on, and on, until a probe is fast, bisect back to the edge, carry on there. A chunk whose drill-down costs over 30 s is condemned whole. What was jumped over is condemned unread; the report says so and --hide-bad cuts it out. Sequential or reverse order only. On under repair
+
when a region is too slow to scan -- most chunks over the last 64 MiB missed their budget, or the scan there, drill-downs and retries included, ran under the --min-rate floor -- stop reading: probe 1 GiB on, and on, until a probe is fast, bisect back to the edge, carry on there. A chunk whose drill-down costs over 10 s is condemned whole. What was jumped over is condemned unread; the report says so and --hide-bad cuts it out. Sequential or reverse order only. On under repair
--no-skip-slow
read every region, however slow, under repair too
--rewrite-weak
diff --git a/hddscan.8 b/hddscan.8 index 245ad1e..7ba6e3d 100644 --- a/hddscan.8 +++ b/hddscan.8 @@ -382,13 +382,13 @@ apiece. A region counts as too slow once three quarters of the chunks over the last 64 MiB have missed their budget, or once scanning it \(em every drill-down, retry and seek included \(em has run below the .B \-\-min\-rate -floor for a minute of I/O. Then stop reading: probe 64 MiB 1 GiB further on, +floor for 20 s of I/O over at least three chunks. Then stop reading: probe 64 MiB 1 GiB further on, and again, until a probe comes back fast, then bisect between the last slow probe and that one down to 16 MiB, with 8 MiB probes once the gap is narrower than a wide one, and carry on scanning there. Every probe is judged against the same calibrated budget as the scan; what drilling a probe's over-budget chunks would cost is taken from what drilling has cost -on this drive so far. Separately, a chunk whose drill-down has spent 30 s of +on this drive so far. Separately, a chunk whose drill-down has spent 10 s of I/O stops being drilled and is condemned whole. What was jumped over is .B condemned without being read: the report says so on its Coverage line and under the verdict, draws it as diff --git a/hddscan.c b/hddscan.c index 797b359..a1359a1 100644 --- a/hddscan.c +++ b/hddscan.c @@ -5375,7 +5375,7 @@ static void check_temperature(ctx_t *c) * budget, and it decides nothing about what is over. * * And whatever the window says, one chunk may not eat the scan: a drill-down - * that has spent 30 s of I/O on one chunk stops, and the whole chunk is + * that has spent 10 s of I/O on one chunk stops, and the whole chunk is * condemned. No window can help while a single chunk takes five minutes. * * Only for an order that walks the drive: sequential or reverse, unsampled. @@ -5392,9 +5392,17 @@ static void check_temperature(ctx_t *c) #define SKIP_WIDE_BYTES (64ull << 20) /* to find damage: 4% of it, 94% */ #define SKIP_PROBE_MIN 4 #define SKIP_PROBE_MAX 512 -#define SKIP_RATE_MIN_CHUNKS 8 /* the rate rule, once it has these */ -#define SKIP_RATE_MIN_US (60ull * 1000000) /* and a minute of I/O */ -#define SKIP_DRILL_CAP_US (30ull * 1000000) /* one chunk's drill-down */ +/* + * How soon. Drilling a healthy 1 MiB chunk costs 256 single reads, about + * 3.5 s on a drive with a 13.6 ms sector median, so a drill-down past 10 s + * of I/O is a chunk the drive is fighting, and three chunks costing 20 s + * between them is a scan running at a twentieth of the floor. Looser than + * this, a drive covering 34 KiB/s spent four minutes before anything could + * be decided -- and a probe that finds nothing costs a second or two. + */ +#define SKIP_RATE_MIN_CHUNKS 3 /* the rate rule, once it has these */ +#define SKIP_RATE_MIN_US (20ull * 1000000) /* and this much I/O */ +#define SKIP_DRILL_CAP_US (10ull * 1000000) /* one chunk's drill-down */ #define MAX_SKIPS 4096 /* the bytes [a, b) covers in the scan order, which only ever walks */ @@ -5537,7 +5545,7 @@ static int skip_cut(ctx_t *c, uint64_t off, size_t len, uint64_t from) /* * One chunk's verdict into the trailing window; 1 when the region it covers * is slow: the window is full and three quarters of it was over budget, or - * scanning it has run below the throughput floor for a minute of I/O. + * scanning it has run below the throughput floor for 20 s of I/O. */ static int skip_note(ctx_t *c, uint64_t step, int over) { @@ -5698,18 +5706,33 @@ static uint64_t skip_ahead(ctx_t *c, void *buf) s0 = c->sw_step[i]; if (c->nskip && s0 < c->skips[c->nskip - 1].s1) s0 = c->skips[c->nskip - 1].s1; - k = skip_add(c); - if (!k) - return t + 1; - k->s0 = s0; - k->k0 = t + 1; - k->s1 = s1; - steps_extent(c, s0, s1, &k->off, &k->len); steps_extent(c, t + 1, s1, &off, &unread); - k->unread = unread; + /* + * Right after a stretch already condemned -- a chunk whose drill-down + * was cut short, most often, which is what set this off -- it is the + * same region, and is written down as one. + */ + if (c->nskip && s0 == c->skips[c->nskip - 1].s1) { + uint64_t was; + + k = &c->skips[c->nskip - 1]; + was = k->len; + k->s1 = s1; + steps_extent(c, k->s0, s1, &k->off, &k->len); + c->skip_cond += k->len - was; + } else { + k = skip_add(c); + if (!k) + return t + 1; + k->s0 = s0; + k->k0 = t + 1; + k->s1 = s1; + steps_extent(c, s0, s1, &k->off, &k->len); + c->skip_cond += k->len; + } + k->unread += unread; skip_band_add(c, off, unread); c->skip_bytes += unread; - c->skip_cond += k->len; memset(c->sw_over, 0, sizeof(c->sw_over)); msg(PROG ": %s: %s%s from %s was too slow to scan; skipped %s " "of it without reading%s (--skip-slow) and carried on %s\n", @@ -7985,7 +8008,7 @@ static void usage(void) " under the --min-rate floor -- stop reading: probe\n" " 1 GiB on, and on, until a probe is fast, bisect\n" " back to the edge, carry on there. A chunk whose\n" -" drill-down costs over 30 s is condemned whole.\n" +" drill-down costs over 10 s is condemned whole.\n" " What was jumped over is condemned unread; the\n" " report says so and --hide-bad cuts it out.\n" " Sequential or reverse order only. On under repair\n" diff --git a/tests/cli.sh b/tests/cli.sh index 8157239..8c2a8d4 100755 --- a/tests/cli.sh +++ b/tests/cli.sh @@ -646,13 +646,13 @@ assert_hasnt "without --skip-slow the rate rule does nothing" \ # And a chunk so full of weak sectors that drilling it alone would take # minutes: every 4 KiB of it reads slow enough to be weak, and retried. -# The drill-down stops once it has cost 30 s, and the chunk is condemned. +# The drill-down stops once it has cost 10 s, and the chunk is condemned. f=$(slowimg skipcut.bin 64M) out=$(HDDSCAN_SLOW=8M:1M HDDSCAN_SLOW_MS=512000 run --chunk 1M \ --chunk-slow-ms 1000 --retries 2 --skip-slow \ --json "$TMP/skipcut.json" "$f") assert_has "a drill-down that costs too much is cut short" "$out" \ - "drilling the chunk at 8.00 MiB has cost over 30 s" + "drilling the chunk at 8.00 MiB has cost over 10 s" read -r roff rend <<< "$(json_region "$TMP/skipcut.json")" assert_eq "and exactly that chunk is condemned" "$roff $rend" \ "8388608 9437184"