Avery Pennarun [Sun, 20 Mar 2011 09:07:03 +0000 (02:07 -0700)]
A bunch of IOError->OSError conversions.
Some of our replacement functions were throwing IOError when the function
they replaced would throw OSError. This was particularly noticeable with
utime() on MacOS, since it caused a unit test failure.
Avery Pennarun [Sun, 20 Mar 2011 08:38:07 +0000 (01:38 -0700)]
metadata: don't die if Linux attr (not xattr) support is missing.
We don't need an import warning for this one, since linuxattr support is
always available on linux, and never available elsewhere, since it's in
_helpers.c and there are no special python modules to install.
Avery Pennarun [Sun, 20 Mar 2011 08:25:13 +0000 (01:25 -0700)]
metadata: recover politely if xattr/acl support is missing.
...previously we'd just crash, which is definitely not polite.
metadata.py now prints warning on import if these features are missing.
That's probably overly obnoxious, especially on systems that don't support
those types of metadata at all. Is there a way to determine whether a
kernel *should* support that type of metadata, so we can warn only if so?
(Obviously if the kernel doesn't support, say, xattrs, there's no point
warning that bup doesn't support them, because no files will be using them
anyway. Hmm...)
Rob Browning [Mon, 14 Mar 2011 01:37:56 +0000 (20:37 -0500)]
Don't accidentally pass atime/ctime/mtime through from_stat_time() twice.
Don't accidentally pass atime/ctime/mtime through
FSTime.from_stat_time() twice in the xstat stat_result.from_stat_rep()
static method when _have_ns_fs_timestamps is false.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Fri, 11 Mar 2011 01:01:30 +0000 (20:01 -0500)]
Makefile: Fix 'clean' rule
In commit 1df0bdd1ad937, I introduced a problem in the make file: the
chmod operation that gives back some permissions on
lib/bup/t/pybuptest.tmp dies if this directory does not exist.
pybuptest.tmp is only created when running the tests.
when the chmod dies, the clean rule stops, thus not completing the
cleanup, so we must make sure this operation is not fatal if the
directory doesn't exist.
Dickon Reed [Fri, 18 Mar 2011 19:25:38 +0000 (12:25 -0700)]
Allow chown to uid:0 to succeed.
The test case assumed that it was not possible to set uid:0 on a file,
which is to say that the current user is not a member of group
0. That's an environmental assumption which is not universal (I am a
counterexample).
Gabriel Filion [Thu, 10 Mar 2011 20:41:54 +0000 (12:41 -0800)]
Verify permissions in check_repo_or_die()
Currently, if one doesn't have read or access permission up to
repo('objects/pack'), bup exits with the following error:
error: repo() is not a bup/git repository
(with repo() replaced with the actual path).
This is misleading, since there is possibly really a repository there
but the user can't access it.
Make git.check_repo_or_die() verify that the current user has the
permission to access repo('objects/pack'), and if not, output a
meaningful error message.
As a bonus, we get an error if the bup_dir path is not a directory.
Gabriel Filion [Mon, 21 Feb 2011 16:14:38 +0000 (11:14 -0500)]
Verify permissions in check_repo_or_die()
Currently, if one doesn't have read or access permission up to
repo('objects/pack'), bup exits with the following error:
error: repo() is not a bup/git repository
(with repo() replaced with the actual path).
This is misleading, since there is possibly really a repository there
but the user can't access it.
Make git.check_repo_or_die() verify that the current user has the
permission to access repo('objects/pack'), and if not, output a
meaningful error message.
As a bonus, we get an error if the bup_dir path is not a directory.
Avery Pennarun [Mon, 28 Feb 2011 09:47:15 +0000 (01:47 -0800)]
Merge branch 'rlb/meta'
* rlb/meta:
t/test-meta.sh: replace 'diff -u5' with 'diff -U5'.
Don't touch correct target xattrs; remove inappropriate target xattrs.
Rename test-fs.img to testfs.img and add it to the clean target.
t/test-meta.sh: detect and handle fakeroot.
Add atime tests and fix atime capture in metadata.from_path().
Improve test-meta.sh status messages.
Handle missing files more gracefully in "bup xstat".
Add initial (trivial) root-only ACL metadata tests and fix exposed bugs.
Add initial (trivial) root-only metadata tests for attr and xattr.
Don't specify 0700 permissions when creating initial directories.
Fix "meta extract -v" directory output.
Fix _apply_common_rec() symlink chown/chmod guard.
Change os.geteuid to os.geteuid() in tmetadata.py.
Remove redundant call to get_linux_file_attr() in _add_linux_attr().
In _add_linux_attr(), catch IOError, not EnvironmentError; handle ENOTTY.
Improve some metadata error messages.
Only print secs for bup xstat times when ns == 0.
Use oct() rather than hex() when printing mode from bup xstat.
Remove bup: prefix from metadata error messages.
Don't "chmod 000" paths during restore.
Remove MetadataError and make apply error handling finer-grained.
Remove MetadataAcquireError and make error handling finer-grained
Accommodate missing owner or group name during metadata save/restore.
Preserve existing stack trace when throwing MetadataErrors.
Add (private for now) "bup xstat" command and use it in the metadata tests.
Also check defined(_ATFILE_SOURCE) in utimensat() guard.
Rename bup-meta.1.md to bup-meta.md.
Simplify FSTime() - always use an integer ns internal representation.
Rename metadata exceptions and add a parent MetadataError class.
Don't use str(e) when instantiating Metadata exceptions.
Fix typos in Metadata._encode_linux_xattr().
Fix handling of conditional definitions in xstat.
Always define _have_ns_fs_timestamps (True or False).
Change "bup meta" to use recursive_dirlist() to add support for --xdev.
Fix minor bug in "bup meta -t" argument handling (if -> elif).
Modify drecurse.py and index.py to use xstat functions.
Move stat-related work to bup.xstat; add xstat.stat.
Add helpers.fstat and _helpers._have_ns_fs_timestamps.
Add a helpers.FSTime class to handle filesystem timestamps and use it.
Attempt to unlink or rmdir existing paths as appropriate during restore.
Conditionalize build/use of get_linux_file_attr and set_linux_file_attr.
Check stat() after attempted restore of nonexistent owner/group in tests.
Don't try to restore owner unless root; handle nonexistent owner/group.
Add metadata test_restore_restricted_user_group().
Add helpers.detect_fakeroot() and use it in relevant metadata tests.
Defer metadata aquisition and application errors during create/extract.
Rename py_* functions to bup_* in lib/bup/_helpers.c.
Don't allow negative ns in metadata timestamps; normalize on read/write.
Add (sec, ns) timestamps and extended stat, lstat, utime, and lutime.
Add vint tests and signed vint support via write_vint and read_vint.
Change user to the more accurate owner in metadata.py.
Correctly respect restore_numeric_ids in Metadata _apply_common_rec().
Send bup meta --list output to stdout, not stderr.
Fix bup-meta.1 start-extract/finish-extract example.
Use Py_RETURN_TRUE in py_lutimes() and py_set_linux_file_attr().
t/test.sh: fix whitespace problems with the 'Inode:' line from 'stat'.
t/test.sh: fix occasional atime-related failure in metadata tests.
t/test.sh: refactoring to reduce duplicated code.
Add initial support for metadata archives.
Avery Pennarun [Mon, 28 Feb 2011 09:33:09 +0000 (01:33 -0800)]
Merge branch 'master' into meta
* master:
midx/bloom: use progress() and debug1() for non-critical messages
helpers: separately determine if stdout and stderr are ttys.
cmd/newliner: restrict progress lines to the screen width.
hashsplit: use shorter offset-filenames inside trees.
Replace 040000 and 0100644 constants with GIT_MODE_{TREE,FILE}
git.py: rename treeparse to tree_decode() and add tree_encode().
hashsplit.py: remove PackWriter-specific knowledge.
cmd/split: fixup progress message, and print -b output incrementally.
hashsplit.py: convert from 'bits' to 'level' earlier in the sequence.
hashsplit.py: okay, *really* fix BLOB_MAX.
hashsplit.py: simplify code and fix BLOB_MAX handling.
options.py: o.fatal(): print error after, not before, usage message.
options.py: make --usage just print the usage message.
When applying xattr metadata, make sure to remove any target xattrs
that aren't in the metadata record, but don't touch target xattrs that
already match the metadata record. Add corresponding tests.
Throw an ApplyError() if xattr set() or remove() return EPERM from
within _apply_linux_xattr_rec().
Remove lib/bup/t/testfs and lib/bup/t/testfs.img in the clean target.
* commit '6f02181':
helpers: separately determine if stdout and stderr are ttys.
cmd/newliner: restrict progress lines to the screen width.
hashsplit: use shorter offset-filenames inside trees.
Replace 040000 and 0100644 constants with GIT_MODE_{TREE,FILE}
git.py: rename treeparse to tree_decode() and add tree_encode().
hashsplit.py: remove PackWriter-specific knowledge.
cmd/split: fixup progress message, and print -b output incrementally.
hashsplit.py: convert from 'bits' to 'level' earlier in the sequence.
hashsplit.py: okay, *really* fix BLOB_MAX.
hashsplit.py: simplify code and fix BLOB_MAX handling.
options.py: o.fatal(): print error after, not before, usage message.
options.py: make --usage just print the usage message.
Gabriel Filion [Fri, 25 Feb 2011 16:16:05 +0000 (11:16 -0500)]
midx/bloom: use progress() and debug1() for non-critical messages
Some messages in these two commands indicate progress but are not
filtered out when the command is not run under a tty. This makes bup
return some unwanted messages when run under cron.
Using progress() and debug1() instead should fix that.
(Changed a few from progress() to debug1() by apenwarr.)
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Avery Pennarun [Sun, 20 Feb 2011 05:21:45 +0000 (21:21 -0800)]
helpers: separately determine if stdout and stderr are ttys.
Previously we only cared if stderr was a tty (since we use that to determine
if we should print progress() or not). But we might want to check stdout as
well, for the same reason that gzip does: we should be refusing to write
binary data to a terminal.
Avery Pennarun [Sun, 20 Feb 2011 02:48:06 +0000 (18:48 -0800)]
hashsplit: use shorter offset-filenames inside trees.
We previously zero-padded all the filenames (which are hexified versions of
the file offsets) to 16 characters, which corresponds to a maximum file size
that fits into a 64-bit integer. I realized that there's no reason to
use a fixed padding length; just pad all the entries in a particular tree to
the length of the longest entry (to ensure that sorting
alphabetically is still equivalent to sorting numerically).
This saves a small amount of space in each tree, which is probably
irrelevant given that gzip compression can quite easily compress extra
zeroes. But it also makes browsing the tree in git look a little prettier.
This is backwards compatible with old versions of vfs.py, since vfs.py has
always just treated the numbers as an ordered set of numbers, and doesn't
care how much zero padding they have.
Avery Pennarun [Sun, 20 Feb 2011 02:02:12 +0000 (18:02 -0800)]
Replace 040000 and 0100644 constants with GIT_MODE_{TREE,FILE}
Those constants were scattered in *way* too many places. While we're there,
fix the inconsistent usage of strings vs. ints when specifying the file
mode; there's no good reason to be passing strings around (except that I
foolishly did that in the original code in version 0.01).
Avery Pennarun [Sun, 20 Feb 2011 01:57:48 +0000 (17:57 -0800)]
git.py: rename treeparse to tree_decode() and add tree_encode().
tree_encode() gets most of its functionality from PackWriter.new_tree(),
which is not just a one liner that calls tree_encode(). We will soon want
to be able to calculate tree hashes without actually writing a tree to a
packfile, so let's split out that functionality.
Let's use callback functions explicitly instead of passing around special
objects; that makes the dependencies a bit more clear and hopefully opens
the way to some more refactoring for clarity.
Avery Pennarun [Thu, 17 Feb 2011 11:10:23 +0000 (03:10 -0800)]
cmd/split: fixup progress message, and print -b output incrementally.
As a side effect, you can no longer combine -b with -t, -c, or -n. But that
was kind of a pointless thing to do anyway, because it silently enforced
--fanout=0, which is almost certainly not what you wanted, precisely if you
were using -t, -c, or -n.
Avery Pennarun [Sun, 20 Feb 2011 04:33:36 +0000 (20:33 -0800)]
hashsplit.py: okay, *really* fix BLOB_MAX.
In some conditions, we were still splitting into blobs larger than BLOB_MAX.
Fix that too.
Unfortunately adding an assertion about it in the 'bup split' main loop
slows things down by a measurable amount, so I can't easily add that to
prevent this from happening by accidenta again in the future.
After implementing this, it looks like 8192 (typical blob size) times two
isn't big enough to prevent this from kicking in in "normal" cases; let's
use 4x instead. In my test file, we exceed this maximum much less. (Every
time we exceed BLOB_MAX, it means the bupsplit algorithm isn't working, so
we won't be deduplicating as effectively. So we want that to be rare.)
Avery Pennarun [Thu, 17 Feb 2011 09:56:31 +0000 (01:56 -0800)]
hashsplit.py: simplify code and fix BLOB_MAX handling.
This reduces the number of lines without removing functionality. I renamed
a few constants to make more sense.
The only functional change is that BLOB_MAX is now an actual maximum instead
of a variable number depending on buf.used(). Previously, it might have
been as large as BLOB_READ_SIZE = 1MB, which is much larger than BLOB_MAX =
16k. Now BLOB_MAX is actually the max.
Avery Pennarun [Sun, 20 Feb 2011 05:37:16 +0000 (21:37 -0800)]
options.py: o.fatal(): print error after, not before, usage message.
git prints the error *before* the usage message, but the more I play with
it, the more I'm annoyed by that behaviour. The usage message can be pretty
long, and the error gots lost way above the usage message. The most
important thing *is* the error, so let's print it last.
Avery Pennarun [Sat, 19 Feb 2011 23:34:45 +0000 (15:34 -0800)]
Merge remote branch 'origin/master' into meta
* origin/master:
doc/import-rsnapshot: small corrections and clarification
cmd/midx, git.py: all else being equal, delete older midxes first.
t/test.sh: a test for the recently-uncovered midx4 problem.
_helpers.c: midx4 didn't handle multiple index with the same object.
cmd/midx: add a --check option.
Add git.shorten_hash(), printing only the first few bytes of a sha1.
tclient.py: add some additional tests that objcache.refresh() is called.
cmd/server: add a debug message saying which object caused a suggestion.
cmd/list-idx: a quick tool for searching the contents of idx/midx files.
Add tests around the bloom ruin and check options
Add a bloom --ruin for testing failure cases
One more constant for header lengths
Split PackMidx from git.py into a new midx.py.
bloom.py: move bloom.ShaBloom.create to just bloom.create.
Move bloom-related stuff from git.py to a new bloom.py.
cmd/bloom: add a --force option to forget regenerating the bloom.
Use the new qprogress() function in more places.
Bail out immediately instead of redownloading .idx
Add a --check behavior to verify bloom
Defines/preprocessor lengths > magic numbers
cmd/{bloom,midx}: clean up progress messages.
cmd/bloom: by default generate bloom filters in *all* index-cache dirs.
cmd/newliner: avoid printing a blank line if the final output ended in \r.
cmd/index: make the default mode '-u'.
_helpers.c: don't cygwin doesn't set any win32 defines.
_helpers.c: don'g unpythonize_argv() on win32.
Remove .c and .o rules, apply flags to csetup.py
Fix a valid warning that some compilers warned
Move .idx file writing to C
main.py: fix whitespace in the usage string.
cmd/daemon: FD_CLOEXEC the listener socket and don't leak fd for the connection.
cmd/daemon: close file descriptors correctly in parent process.
cmd/daemon: use SO_REUSEADDR.
cmd/daemon: pass extra options along to 'bup server'.
cmd/daemon: correctly report socket binding/listening errors.
main.py: use execvp() instead of subprocess.Popen() when possible.
_helpers.c: Remove ugly 'python' junk from argv[0] so 'ps' is prettier.
cmd/bloom: fix a message pluralization.
cmd/join: add a new -o (output filename) option.
cmd/ls: fix a typo causing 'bup ls foo/latest' to not work.
cmd/server: add a new 'help' command.
midx4: Fix the other side of my previous nasty bug
Gabriel Filion [Fri, 18 Feb 2011 19:15:41 +0000 (14:15 -0500)]
doc/import-rsnapshot: small corrections and clarification
There's a typo in the --dry-run option explanation.
The form "[...] or only imports all [...]" is confusing. Turn it around
a little bit so that the quantifiers are associated more easily to the
right portions of the sentence.
Also, add an example for using the backuptarget argument.
Avery Pennarun [Fri, 18 Feb 2011 09:15:47 +0000 (01:15 -0800)]
cmd/midx, git.py: all else being equal, delete older midxes first.
Previous runs of 'bup midx -f' might have created invalid midx files with
exactly the same length as a newer run. bup's "prune redundant midx" logic
would quasi-randomly choose one or the other to delete (based on
alphabetical order of filenames, basically) and sometimes that would be the
new one, not the old one, so the 'bup midx -f' results never actually kicked
in.
Now if the file sizes are equal we'll use the mtime as a tie breaker; newer
is better.
Avery Pennarun [Fri, 18 Feb 2011 08:44:32 +0000 (00:44 -0800)]
_helpers.c: midx4 didn't handle multiple index with the same object.
It *tried* to handle it, but would end up with a bunch of zero entries at
the end, which prevents .exists() from working correctly in some cases.
In midx2, it made sense to never include the same entry twice, because the
only informatin we had about a particular entry was that it existed. In
midx4 this is no longer true; we might sometimes want to know *all* the idx
files that contain a particular object (for example, when we implement
expiry later). So the easiest fix for this bug is to just include multiple
entries when we have them.
Avery Pennarun [Thu, 17 Feb 2011 02:39:35 +0000 (18:39 -0800)]
bloom.py: move bloom.ShaBloom.create to just bloom.create.
I don't really like class-level functions. Ideally we'd just move all the
creation stuff into cmd/bloom, but tbloom.py is testing them, so it's not
really worth it.
Avery Pennarun [Thu, 17 Feb 2011 02:05:18 +0000 (18:05 -0800)]
Move bloom-related stuff from git.py to a new bloom.py.
No other functionality changes other than that cmd/memtest now reports the
number of bloom steps separately from the midx/idx steps. (This is mostly
so they don't have to share the same global variables, but it's also
interesting information to break out.)
Avery Pennarun [Thu, 17 Feb 2011 00:11:26 +0000 (16:11 -0800)]
Use the new qprogress() function in more places.
qprogress() was introduced in the last commit and has smarter default
behaviour that automatically reduces progress output so we don't print too
many messages per second. Various commands/etc were doing this in various
different ad-hoc ways, but let's centralize it all in one place.
Brandon Low [Thu, 17 Feb 2011 01:46:02 +0000 (17:46 -0800)]
Add a --check behavior to verify bloom
This new behavior is useful when diagnosing weird behavior, lets a bloom
filter claiming to contain a particular idx be verified against that idx
file.
Avery Pennarun [Thu, 17 Feb 2011 00:01:30 +0000 (16:01 -0800)]
cmd/{bloom,midx}: clean up progress messages.
bloom was printing messages more often than necessary on fast computers,
which could overwhelm the stderr output a bit. Also change to a percentage
+ number of objects, like midx and save do, rather than just printing the
current file number.
And don't print so many lines of output by default: now if bloom isn't
end up doing anything, it doesn't print any output. And if it does do
something, it prints only one output line per file.
bloom and midx now both print the name of the directory where they're
creating their output files; if you have multiple directories in
.bup/index-cache, it was a little confusing to see them doing
multiple runs for no apparent reason.
Avery Pennarun [Wed, 16 Feb 2011 23:58:14 +0000 (15:58 -0800)]
cmd/bloom: by default generate bloom filters in *all* index-cache dirs.
This matches with 'bup midx -a' and 'bup midx -f' behaviour. People might
have been thinking they were regenerating bloom filters without actually
doing them all.
Moved some shared code to do this from cmd/midx to git.py.
Avery Pennarun [Wed, 16 Feb 2011 23:55:24 +0000 (15:55 -0800)]
cmd/newliner: avoid printing a blank line if the final output ended in \r.
If the last output was a progress message, we would blank out the line
(which was correct) but then we'd print a newline, which was wrong. Only
print the leftover output, followed by a newline, if the last output was
nonempty.
Avery Pennarun [Wed, 16 Feb 2011 23:14:01 +0000 (15:14 -0800)]
cmd/index: make the default mode '-u'.
I always forget the -u option, and it's by far the most common thing to do
with 'bup index'. So if no mode option is provided, just default to that
one.
While we're here, update the man page and usage message a bit.
Brandon Low [Mon, 14 Feb 2011 19:27:29 +0000 (11:27 -0800)]
Remove .c and .o rules, apply flags to csetup.py
The .c and .o rules were not used and were misleadingly implying that we
were already paying attention to LDFLAGS and CFLAGS. Instead apply the
flags to csetup.py where they will actually do something.
Brandon Low [Sun, 13 Feb 2011 19:17:06 +0000 (11:17 -0800)]
Move .idx file writing to C
This was a remaining CPU bottleneck in bup-dumb-server mode. In a quick
test, writing 10 .idx files of 100000 elements on my netbook went from
50s to 4s. There may be more performance available by adjusting the
definition of the PackWriter.idx object, but list(list(tuple)) isn't
bad.
Avery Pennarun [Sun, 13 Feb 2011 12:10:08 +0000 (04:10 -0800)]
cmd/daemon: FD_CLOEXEC the listener socket and don't leak fd for the connection.
Otherwise the listener gets inherited by all the child processes (mostly
harmless) and subprograms run by bup-server inherit an extra fd for the
connection socket (problematic since we want the connection to close as soon
as bup-server closes).
Avery Pennarun [Sun, 13 Feb 2011 10:29:59 +0000 (02:29 -0800)]
cmd/daemon: close file descriptors correctly in parent process.
The client wasn't getting disconnected when the server died, because the
daemon was still hanging on to its copy of the original socket, due to some
misplaced os.dup() calls.
Avery Pennarun [Sun, 13 Feb 2011 10:18:57 +0000 (02:18 -0800)]
cmd/daemon: pass extra options along to 'bup server'.
Currently 'bup server' doesn't take any options, but that might change
someday.
Also use a '--' to separate the bup mux command from its arguments, so it
doesn't accidentally try to parse them. This didn't matter before (since
none of the options we were passing along started with a dash) but if the
user provides extra options, it might matter.
We should never, ever throw away the string from an exception, because
that's how people debug problems. (In this case, my problem was "address
already in use.")
Avery Pennarun [Sun, 13 Feb 2011 09:50:50 +0000 (01:50 -0800)]
_helpers.c: Remove ugly 'python' junk from argv[0] so 'ps' is prettier.
Okay, this is pretty gross. But the 'ps' output was looking ugly, and
now it doesn't. We remove the 'python' interpreter string and the expanded
pathname of the command being run, so it now shows as (eg.) "bup-join" instead
of "python /blah/blah/blah/cmd/bup-join".
Brandon Low [Thu, 10 Feb 2011 21:23:36 +0000 (13:23 -0800)]
midx4: Fix the other side of my previous nasty bug
The previous one was a problem with midx4s generated from idx files,
this one is similar but when they are generated from other .midx4 files.
Many thanks to Aneurin Price for putting up with the awful behavior and
prodding at bup and whatnot while I was trying to make this one
disappear under a rug.
Once again, midx4 files generated prior to this patch will want to be
regenerated. Once again, only smart servers which have objects not on
the client's index cache will be effected, but they sure as hell well be
effected.
Brandon Low [Tue, 8 Feb 2011 18:43:22 +0000 (10:43 -0800)]
midx4: Fix name offsets when generated from idx
This was a nasty bug, glad it got found before release. Only effected
the server's ability to suggest .idxs so far, but would have effected
any attempt to have bup retrieve objects directly too.
Avery Pennarun [Tue, 8 Feb 2011 06:16:08 +0000 (22:16 -0800)]
Merge branch 'bloom'
* bloom:
bloom: avoid kernel disk flushes when we dirty a lot of pages.
midx4: Properly decide whether to do progress in C
midx4: Don't use Py_ssize_t, it's not in python2.4
cmd/bloom: map only one .idx file at a time.
bloom: Use truncate not writing zeros in create
bloom: Don't use function pointers in tight loops
Fix updating of bloom with additional files
ShaBloom.init(): initialize members before the assert().
cmd/bloom: actually, always use the same temp filename.
cmd/bloom: use mkstemp() instead of NamedTemporaryFile().
midx: Write midx4 in C rather than python
midx4: midx2 with idx backreferences
ShaBloom: Add k=4 support for large repositories
ShaBloom prefilter to detect nonexistant objects
mmap: Make closing source file optional
Avery Pennarun [Tue, 8 Feb 2011 03:09:06 +0000 (19:09 -0800)]
bloom: avoid kernel disk flushes when we dirty a lot of pages.
Based on the number of objects we'll add to the bloom, decide if we want to
mmap() the pages as shared-writable ('immediate' write) or else map them
private-writable for later manual writing back to the file ('delayed'
write).
A bloom table's write access pattern is such that we dirty almost all the
pages after adding very few entries; essentially, we can expect to dirty
about n*k/4096 pages if we add n objects to the bloom with k hashes. But the
table is so big that dirtying *all* the pages often exceeds Linux's default
/proc/sys/vm/dirty_ratio or /proc/sys/vm/dirty_background_ratio,
thus causing it to start flushing the table before we're
finished... even though there's more than enough space to
store the bloom table in RAM.
To work around that behaviour, if we calculate that we'll probably end up
touching the whole table anyway (at least one bit flipped per memory page),
let's use a "private" mmap, which defeats Linux's ability to flush it to
disk. Then we'll flush it as one big lump during close(), which doesn't
lose any time since we would have had to flush all the pages anyway.
While we're here, let's remove the readwrite=True option to
ShaBloom.create(); nobody's going to create a bloom file that isn't
writable.
Brandon Low [Tue, 8 Feb 2011 02:30:04 +0000 (18:30 -0800)]
midx4: Properly decide whether to do progress in C
Basically just gives us a _helpers.istty to go along with helpers.istty
and uses it to decide whether or not to write progress messages from
midx4 generation.
Avery Pennarun [Tue, 8 Feb 2011 01:41:00 +0000 (17:41 -0800)]
cmd/bloom: map only one .idx file at a time.
This massively decreases virtual memory allocation since we only ever need
to look at a single idx at once.
In theory, VM doesn't cost us anything, but on 32-bit systems we can
actually run out of address space if we try to map all the idx files at
once on a very large repo.
Brandon Low [Mon, 7 Feb 2011 17:08:00 +0000 (09:08 -0800)]
bloom: Use truncate not writing zeros in create
This lets us test more of bloom's code without writing gigabyte(s) of
zeros to disk. As noted in the NOTE: this works on all of the common
modern unixes that I checked, but may need special handling on other
systems.
projects / bup.git / log