mincore: fix reading information

The _fmincore_chunk_size is typically set to 64MiB, which
makes sense to avoid doing very large mmap() operations
(to save already precious VM on 32-bit systems).

However, since that's in bytes, we cannot divide a size in
pages by it, and expect any useful outcome.

Calculate the number of chunks (chunk_count) properly based
on the size of the file, rather than its number of pages.
Otherwise, chunk_count typically ends up just 1 even for a
very large file (my test file was ~500MiB), and mincore()
is run just once, so we fill the presence information only
for the first 64MiB of the file, even if it was previously
completely in RAM.

Given a large enough test file (and enough RAM to keep it
there), the following should print about the same times
twice:

  cat test > /dev/null ; \
  time cat test > /dev/null ; \
  bup split --noop test ; \
  time cat test >/dev/null

Without the fix, it's evident that the file is evicted from
RAM almost entirely (apart from the first 64MiB) even in
this synthetic case.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

diff --git a/lib/bup/helpers.py b/lib/bup/helpers.py
index f38435557f29c1642609205e3765a1bd64d5647b..d9886854c091f0f5ff473c386c3446325813063f 100644 (file)
--- a/lib/bup/helpers.py
+++ b/lib/bup/helpers.py
@@ -803,9 +803,7 @@ if _mincore:
             _set_fmincore_chunk_size()
         pages_per_chunk = _fmincore_chunk_size // sc_page_size;
         page_count = (st.st_size + sc_page_size - 1) // sc_page_size;
-        chunk_count = page_count // _fmincore_chunk_size
-        if chunk_count < 1:
-            chunk_count = 1
+        chunk_count = (st.st_size + _fmincore_chunk_size - 1) // _fmincore_chunk_size
         result = bytearray(page_count)
         for ci in compat.range(chunk_count):
             pos = _fmincore_chunk_size * ci;