From: Rob Browning Date: Wed, 21 Aug 2019 06:23:30 +0000 (-0500) Subject: fsck: disable par2's internal concurrency X-Git-Tag: 0.30~14 X-Git-Url: https://arthur.barton.de/cgi-bin/gitweb.cgi?p=bup.git;a=commitdiff_plain;h=24208d7a7cd30b9b5e2838a96b79c36e3f746b1d fsck: disable par2's internal concurrency It looks like par2 added its own concurrency a bit back, and when that's combined with bup's fsck -j parallelism, a simple t/test-fsck.sh run completely swamps a machine here, launching what looks like 32 threads total (for a 4/8 core machine), and fully saturating the CPU. The current test, which specifies -j99 actually ends up launching 4 par2 invocations, each processing a single file, and then each par2 appears to assume it has the entire machine to itself and launches 8 threads (one per hardware "core"). The resulting test takes 100s, but if we disable par2's parallelism with -t1, putting bup's -j argument back in control of the overall level of concurrency, the run time comes down to 4s. Signed-off-by: Rob Browning Tested-by: Rob Browning --- diff --git a/cmd/fsck-cmd.py b/cmd/fsck-cmd.py index 35b45f8..8fc4f7f 100755 --- a/cmd/fsck-cmd.py +++ b/cmd/fsck-cmd.py @@ -52,14 +52,14 @@ def parv(lvl): return ['-qq'] def par2_generate(base): - return run(['par2', 'create', '-n1', '-c200'] + parv(2) + return run(['par2', 'create', '-t1', '-n1', '-c200'] + parv(2) + ['--', base, base+'.pack', base+'.idx']) def par2_verify(base): - return run(['par2', 'verify'] + parv(3) + ['--', base]) + return run(['par2', 'verify', '-t1'] + parv(3) + ['--', base]) def par2_repair(base): - return run(['par2', 'repair'] + parv(2) + ['--', base]) + return run(['par2', 'repair', '-t1'] + parv(2) + ['--', base]) def quick_verify(base): f = open(base + '.pack', 'rb')