Johannes Berg [Sat, 25 Jan 2020 22:13:17 +0000 (23:13 +0100)]
tests: add test for save encountering duplicates
Add a test for save encountering duplicates in the index, both
for a file and a directory, which was fixed in the previous patch.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit bfc9e621ee5801fd597b3668a0a7f6747b7cba3a) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 22:41:56 +0000 (23:41 +0100)]
save: don't confuse metadata on duplicate files
If there are duplicate files, save removes the duplicates but
doesn't remove the duplicates from metadata, so the metadata
list gets messed up wrt. the file list. Fix this.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit bd0dc75dd161d544133cc80d4858bc8ef1511a7e)
[rlb@defaultvalue.org: adapt to 0.3x] Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:21:24 +0000 (22:21 +0100)]
test-save-errors: add tests for inaccessible metadata
Add test cases for when metadata for a file and directory cannot
be read (causing an IOError instead).
Note that this test fails if the previous two patches aren't
applied.
Note also that if such an error ever happened without the patch
to save, the repository is essentially corrupt and there no way
to figure out _which_ file didn't get metadata.
Note that this uses some python trickery to force an IOError in
metadata reading.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: name test in commit summary line] Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 19b3b1b7acda5e507a4ec053cff6a2e139c24f31)
[rlb@defaultvalue.org: adjust bup-save test overrides for 0.3x] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:20:36 +0000 (22:20 +0100)]
vfs: deal with bupm containing no metadata for a file
If a file was for some reason saved without metadata, deal with
this and create an empty/unknown metadata for it instead.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit da552ea8d19d5da73c0d387d4963bc77841ab9ba) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:18:53 +0000 (22:18 +0100)]
save: add empty metadata if reading fails
If we cannot read a file's metadata, we must add an entry in the
repository nonetheless because the bupm contents must match the
tree contents.
Add a dummy (empty) metadata entry in this case, as is already
done when reading a directory's metadata.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 47891d8951a95b8e0d9ca94387107cdf12ca3d3c) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:20:09 +0000 (00:20 +0100)]
index: fix Entry comparison methods
Some of these methods are evidently not used today,
but they should have proper arguments anyway.
Fixes: 3ff7455dd474 ("Don't use cmp()") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit c7013fde92fc73e0064462954918b09541e9a84f) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 20:40:31 +0000 (21:40 +0100)]
test-web: check that static/styles.css can be retrieved
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 188ccbafbceed4795fb7dddbce97d1acccca9242) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket
Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 21fbd0c733fe2abdeddcf7a5aaf7bab85890de2c) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi
Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 041082521f8bdc46e6a36347a9f5183abbe4f272) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:33 +0000 (00:22 +0100)]
client: import DemuxConn
Fixes: fb3bd84cfd24 ("Add DemuxConn and `bup mux` for client-server") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 1ad93eca8d0e837263d9f29dd229f6835f452c04) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:19:33 +0000 (00:19 +0100)]
vfs: fix finish_extract()
The 'dir' variable doesn't exist here, must be 'meta' instead.
Fixes: 0962d3904735 ("Add initial support for metadata archives.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 9cbb51ec1423ef2e24f18f756585dcc39d8a54f2) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 18 Dec 2019 21:43:58 +0000 (22:43 +0100)]
get: remove extra src_repo
We already have a src_repo from the with statement, no need to
instantiate another one (that won't even be closed properly).
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit ab051b4620832ef8e2b4dd134b1b61241f53814e) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 2 Jan 2020 21:34:52 +0000 (22:34 +0100)]
tests: fuse: use stat instead of relying on "ls -l" output
On my Fedora 31 system, this currently fails because the test
expects the file mode bits to be printed as "-rw-r--r--" but
I get "-rw-r--r--." instead, perhaps due to selinux.
Fix this to use stat --format instead to extract the bits we
want to check here (user, group, mode, timestamp).
While at it, I noticed that this test has some confusion with
timezones and only worked because 'ls -l' doesn't show the
time in this case since the file is so old. Fix that by making
_everything_ here use UTC, instead of just some parts.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit c38e02f0f5daa4ddf4d2c0a553d717d1f3ad02e5) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 13 Dec 2019 20:37:16 +0000 (21:37 +0100)]
man pages: fix some formatting
I noticed one place where -- was used instead of \--, leading to a
emdash instead of two hyphens, but then I looked and realized this
was not just a single place; fix all of those.
While at it, I saw some --keep options being referenced without
backticks, fix that too.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit 27e6ee3680585fd445ce04955333a4bdf585db30) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 11 Feb 2020 02:47:13 +0000 (20:47 -0600)]
git: overhaul git version checking
Rework the git version tetsting to handle versions like "1.5.2-rc3"
or (apparently) "1.5.2-rc3 (something ...)". Add tests for parsing of
all the version types in the current git tag history that we need to
support.
Support and document BUP_ASSUME_GIT_VERSION_IS_FINE=1 as an escape
hatch in case the parsing isn't sufficiently comprehensive, or
upstreeam changes their practices in future releases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit c9227455fdbd6605c04075aa2ae0faad9aacbc62) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 13 Dec 2019 20:36:04 +0000 (21:36 +0100)]
CatPipe: remove _ver_warned
This variable is no longer used, remove it entirely
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit bcdc1c7ff1ba166aa1990f77d71bb410cae959c3) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 12 Jan 2020 21:33:15 +0000 (15:33 -0600)]
bloom: fix logic controlling bloom regeneration
Add missing MAX_BLOOM_BITS index in the logic in bup bloom that
determines whether or not we should regenerate the filter. We never
noticed because:
$ python2
>>> 0 < {1 : 2}
True
$ python3
>>> 0 < {1 : 2}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'dict'
Also regnerate if the -k value differs from the existing filter's k.
Thanks to Johannes Berg for pointing out some nontrivial problems in
an earlier version.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit b86fb21ae670f8d1dc42d26bd69252cb9cc0ab7f) Tested-by: Rob Browning <rlb@defaultvalue.org>
The Python documentation [0] indicates that an 'O' passed to
Py_BuildValue will have its refcount incremented. Since some elements of
the tuple created in stat_struct_to_py are pre-converted in C to
PyObjects, they already have a refcount of 1 - use 'N' to avoid
incrementing it and ensure Python can deallocate them correctly.
Rob Browning [Sun, 20 Oct 2019 16:48:58 +0000 (11:48 -0500)]
vfs: rely on FakeLink's target in readlink
This could result in an error like:
File "/usr/local/lib/bup/bup/vfs.py", line 524, in _compute_item_size
return len(_readlink(repo, item.oid))
AttributeError: 'FakeLink' object has no attribute 'oid'
Thanks to Hartmut Krafft for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 023917c97cbf9bf6dbfad7120b41850dc5d7c7cd) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 11 Sep 2019 02:54:15 +0000 (21:54 -0500)]
test-get: rm pax_global_header from git archive output
git-archive now adds a pax_global_header, and while GNU tar suppresses
it, some versions of tar don't, so remove it explicitly to avoid
spurious test failures.
Thanks to Greg Troxel for for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 21:02:18 +0000 (16:02 -0500)]
cirrus-ci: specify make -j3 on all platforms
Based on past experience, this should help -- locally it has appeared
to help our tests even with a greater concurrency level than the cpu
count, and it looks like all the current test hosts have at least two
cores.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 18:18:17 +0000 (13:18 -0500)]
prep-for-macos-build: get brew rsync
It looks like macos may ship with a more limited rsync (or at least
one that doesn't produce the --itemize-changes output we expect, so
just install brew's.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 16:49:54 +0000 (11:49 -0500)]
fsck: only specify -t1 when par2 appears to support it
It looks like par2 may reject the "-t1" argument even when it's new
enough to support parallelism, so we can't rely on detecting the
version. Instead, before running the first real par2 command, test
-t1 in a sandbox to decide whether we can use it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 21 Aug 2019 06:23:30 +0000 (01:23 -0500)]
fsck: disable par2's internal concurrency
It looks like par2 added its own concurrency a bit back, and when
that's combined with bup's fsck -j parallelism, a simple
t/test-fsck.sh run completely swamps a machine here, launching what
looks like 32 threads total (for a 4/8 core machine), and fully
saturating the CPU.
The current test, which specifies -j99 actually ends up launching 4
par2 invocations, each processing a single file, and then each par2
appears to assume it has the entire machine to itself and launches 8
threads (one per hardware "core").
The resulting test takes 100s, but if we disable par2's parallelism
with -t1, putting bup's -j argument back in control of the overall
level of concurrency, the run time comes down to 4s.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 20 Aug 2019 01:34:11 +0000 (20:34 -0500)]
test-prune-older: disable background gc
The debian buildd's exposed a race between the default automatic
background gc and the explict gc in the test, resulting in:
git commit --date 1566025987 -qam 1566025987
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
git gc --aggressive
fatal: gc is already running on machine 'zandonai' pid 34323 (use --force if not)
Traceback (most recent call last):
File "t/test-prune-older", line 190, in <module>
save_utcs = create_older_random_saves(save_population, three_years_ago, now)
File "t/test-prune-older", line 74, in create_older_random_saves
exc(['git', 'gc', '--aggressive'])
File "t/test-prune-older", line 41, in exc
check_call(cmd, shell=shell)
File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'gc', '--aggressive']' returned non-zero exit status 128
Fix it by disabling gc.autoDetach in the git config for the test repo
-- something we may well need to do in other tests too.
Thanks to Robert Edmonds for reporting the problem and suggesting the
fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
If the watched process ends a push to the pipe without a newline at
the end, but with newlines in the middle, then sep_rx.split() will
return with multiple entries, the last of which will not end with a
newline and yet not be the empty string. This line prefix needs to be
stashed into the pending buffer, too.
This turns out to be exactly the same logic as if sep_rx.split had not
split the string, so eliminate one layer of conditionals.
This version incorporates feedback from Rob Browning to continue to
pass a list to extend().
Signed-off-by: Nathaniel Filardo <nwf20@cl.cam.ac.uk>
[rlb@defaultvalue.org: adjust commit summary and remove extra space in
"if split[0]" guard.] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Greg Troxel [Wed, 5 Jun 2019 20:55:32 +0000 (16:55 -0400)]
restore: create fifos with mkfifo, not mknod
I recently did a restore of a large bup backup, about 34G worth. All
worked well, including metadata, except that bup threw an exception on
restoring fifos (that I didn't need; they were in /var and were sockets
in use by daemons when the backup happened).
The problem was that mknod was being called for the fifo, and given only
two argumetns. mknod(2) on NetBSD says it takes three arguments.
mkfifo takes two. I am guessing that mknod in python calls mknod the OS
call, and on Linux somehow the third null argument works out somehow.
But it seems irregular to make a fifo with mknod.
I realize python is not POSIX, but mknod(2) requires three arguments:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/mknod.html
It would be nice to have a test of backing up and restoring a fifo; that
would have caught this.
The following patch makes my restore go smoothly.
Signed-off-by: Greg Troxel <gdt@lexort.com>
[rlb@defaultvalue.org: adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 13 Apr 2019 17:07:18 +0000 (12:07 -0500)]
rev_list: handle multiple results/ref from remote for custom formats
Previously a remote rev_list invocation would crash if a custom format
was provided and the remote produced multiple results for any input
ref. Change the client/server code to use a blank line to indicate
the end of the rev-list results. Of course, that means that the parse
function provided must be able to handle (consume) any blank lines
that its format string produces, which may preclude the use of some
format strings, but should be sufficient for now.
Adjust test-ls to (trivially) cover this case, and broaden the use of
the commit hash length tests in the code.
Thanks to Alex Roper for reporting the problem, providing an easy way
to reproduce it, and proposing a fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 12 Apr 2019 19:55:13 +0000 (14:55 -0500)]
Handle commit mergetags (at all)
Previously bup would just crash (i.e. bup ls) if it encountered a
commit with a mergetag header (apparently a new thing). For now,
adjust git.parse_commit to accept and ignore them as long as they only
appear as an optional, final header in the commit. That may or may
not turn out to be sufficient, and it does mean that for now we won't
be able to preserve mergetags (if we want to) whenever we rewrite
commits via bup gc, get, etc.
Rob Browning [Sun, 23 Mar 2014 17:41:06 +0000 (12:41 -0500)]
Add bup get; see the documentation for further information
WARNING: this is a new EXPERIMENTAL command that can (intentionally)
modify your data in destructive ways. Treat with caution.
Thanks to Karl Kiniger <karl.kiniger@med.ge.com> for helping track
down various bugs in earlier versions, and for noting that we might
want --verbose to be a little more effusive. And thanks to Patryck
Rouleau <prouleau72@gmail.com> for suggesting improvements to the
documentation.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 20:32:52 +0000 (15:32 -0500)]
repo: avoid cyclic dependency with is_remote method
The current VFS operations (like resolve()) require a repo object, but
we're about to add a VFS resolve() method to the repos. In and of
itself, that isn't necessarily a problem, but as an optimization, we
want the VFS resolve() to be able to detect when the repo it's been
given is a RemoteRepo and redirect the call to remote_repo.resolve().
Doing so pushes the one single resolve() call to the remote instead of
executing the resolve() locally with a lot of individual calls to the
remote_repo's other methods.
Adding is_remote() makes that possible without having to 'import repo'
in the VFS (repo already imports vfs).
Perhaps we'll rework it later, but this will do for now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 19:14:43 +0000 (14:14 -0500)]
Move vfs resolve() tests to tresolve.py
Move resolve() tests from tvfs to tresolve, and the common tree_dict()
test code to a new test.vfs module, in preparation for more extensive
resolve() testing.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Jan 2019 22:22:14 +0000 (16:22 -0600)]
test_resolve_loop: ensure exception is actually thrown
Make sure to resolve the correct path, and ensure the call never
returns. Previously when the path was wrong, and it *was* wrong, the
test would appear to succeed even though it wasn't actually testing
the intended ELOOP case.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 23:04:27 +0000 (18:04 -0500)]
metadata: always add/store/retrieve size for links and normal files
This simplifies cases where we need to transmit Metadata
objects (i.e. bup-get's repo.resolve()), and it means that for trees
created using this new v3 format, retrieving the sizes of chunked
files should be notably less expensive, since they'll be directly
available in the directory's .bupm file.
Without that, we have to seek around in the chunked tree to find the
last byte (cf. vfs._normal_or_chunked_file_size).
Only store the sizes for symlinks and regular files (which might be
chunked) until it's clear that other st_sizes are useful.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Feb 2018 07:32:54 +0000 (23:32 -0800)]
Replace lresolve with resolve(..., follow=False)
Although there's the NOFOLLOW precedent, this is really just to narrow
the API before we add it as a repo method, i.e. so we only have to
handle one function instead of two.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 9 Dec 2018 18:40:26 +0000 (12:40 -0600)]
vfs: change /save/latest back to a symlink to the latest save
The current, reworked vfs presents /somesave/latest as if it actually
is the latest commit. Change it back to a symlink to the latest save
to roughly match the previous behavior -- though now it's a link to
the save name, not to the (removed) /.commit/ subtree.
To restore the link, reintroduce the concept of a fake
symlink (i.e. one that has no corresponding blob in the repository).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 3 Dec 2018 18:58:03 +0000 (12:58 -0600)]
Don't return invalid data for offset reads (observed via fuse)
Fix a serious bug in the vfs that could cause it to return invalid
data for chunked file read()s that didn't start at the beginning of
the file. The issue was first observed via fuse, which makes sense
given that it streams a file in chunks that (currently) each come from
independent, increasing seek-offset FileReaders.
The previous dropwhile() invocation in the _tree_chunks generator,
used to skip past chunks that were completely before the offset, was
simple but wrong, and would skip too far. Replace it with
_skip_chunks_before_offset().
Add randomized tests of both simple streaming reads, and seek offset
reads, which catch the problem, cover additional cases, and should
prevent regressions.
Thanks to voldial for reporting the problem and providing an easy way
to reproduce it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 17:35:11 +0000 (12:35 -0500)]
update-doc-branches: add command to update man and html
Create a new command to update the man and html branches, and move the
related code there from the Makefile.
Update the branches based on the current (clean) tree, rather than
consulting the git origin, and rely on ls-files rather than globbing
so that the file lists will always be correct -- we'll immediately
notice deletions, avoid picking up stray files in the directory, etc.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 30 Sep 2018 20:29:42 +0000 (15:29 -0500)]
Makefile: fix find -printf issue on FreeBSD
Apparently the use of -printf was causing the error: "printf: missing
format character" with FreeBSD 11.1-RELEASE. Change the helpers lib
count to rely on -print0 and tr, which should be more portable, but
still be undisturbed by unusual paths.
Thanks to Curtis Dunham for reporting the problem and proposing an
alternate solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 7 Jul 2018 17:20:47 +0000 (12:20 -0500)]
vfs: flatten resolution cache key
This will require more storage if there are a lot of lookups with the
same parent and differing paths, but otherwise, without more
intentional structure sharing among paths, this should be better, and
we can always revisit the arrangement later.
Serializing the parent path segments should also make sure the same
parent (semantically-speaking) contributes the same hash to the key.
Previously, Metadata objects could prevent that, given their trivial,
pointer-based hashes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 2 Sep 2018 17:29:01 +0000 (12:29 -0500)]
Refuse to run by default if python version is not 2
Exit with an error if the python major version isn't 2, since we're
working on support for py3, and we'll probably reach a point before
we're finished where bup doesn't immediately crash with py3, but might
well do very bad things to the repository.
Allow overriding the check by setting
BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true so people can still test py3
if they like.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 09:57:18 +0000 (19:57 +1000)]
tindex: add 0o to current octal literal
Needed for python3.
This preserves current behaviour, but since I get no test failures if I
just chop of the leading 0 and make it a normal int there's a good
chance there's underlying breakage.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: adjust commit message summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 08:25:25 +0000 (18:25 +1000)]
Replace remaining print statements with print function
Also add the appropriate __future__ import statement.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: memtest-cmd.py: remove a few preexisting
trailing spaces that were carried over; adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Jul 2018 17:01:55 +0000 (12:01 -0500)]
vfs: include unique repo id in resolution cache key
...since resolve() currently requires a full parent path and the root
refs are only applicable to a particular repository.
Use differing integers to identify repositories that may be
independent (with respect to refs, tags, etc.), and use (typically
small) integers rather than the repo path/address so that they'll be
short if we want to embed them directly in cache keys later.
Use realpath() for local repositories in order to detect when the same
repository is reachable by multiple paths. (Something similar could
eventually be done for remotes.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 6 May 2018 16:46:45 +0000 (11:46 -0500)]
vfs: cache resolve() calls to improve (fuse) performance
Include resolve() results in the vfs cache. This substantially
improves fuse "cat somefile" performance. (Observed a ~2x rate
improvement with a 500MB urandom file).
This appears to be due to the fact that fuse read(path, offset, len)
is called many times for the file, resulting in many corresponding,
redundant resolve(path) calls.
The previous fuse implementation, based on the previous vfs had its
own cache, but moving the caching to the vfs should be more generally
helpful.
Now bup fuse will again ignore repository changes that affect paths it
has already examined. This matches its behavior in the current stable
release (0.29.1).
Thanks to voldial for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log