Johannes Berg [Mon, 16 Dec 2019 09:12:24 +0000 (10:12 +0100)]
metadata: use python-xattr pyxattr compat code
Newer versions of python-xattr have pyxattr compatibility code
(since my pull request https://github.com/xattr/xattr/pull/38).
Modify bup to allow using it; prefer pyxattr since it's faster,
but sometimes due to other dependencies python-xattr might be
present already (and since it conflicts with pyxattr, it's not
possible to install both.)
Also add a .cirrus.yml task to test this on Debian.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Fri, 13 Dec 2019 20:37:16 +0000 (21:37 +0100)]
man pages: fix some formatting
I noticed one place where -- was used instead of \--, leading to a
emdash instead of two hyphens, but then I looked and realized this
was not just a single place; fix all of those.
While at it, I saw some --keep options being referenced without
backticks, fix that too.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Rob Browning [Fri, 18 Oct 2019 04:22:59 +0000 (23:22 -0500)]
merge_iter: don't compare generators with < via heapq
Previously we were implicitly comparing (entry, reader_generator) with
"<" in the merge_iter heapq. In python 2, the generators would just
compare as pointers, which was fine, because we only cared about the
ordering of the initial entries.
However, in Python 3 comparing generators provokes a TypeError, so
create a trivial helper class that can carry the two items, but only
compare the entries via the < operator.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 14 Oct 2019 23:48:24 +0000 (18:48 -0500)]
Make adjustments to fix tbloom for python 3
Make the changes needed to get the tbloom tests passing with python 3.
Change the C helper argument conversions to types that work with each
version (there didn't appear to be a good, shared common format string
type). Switch to a Py_buffer in some places because python 3 didn't
appear to have a conversion that would convert all the binary
inputs (buffer/memoryview/...) to a plain C byte array the way python
2 can.
Change all the other typical things, i.e. byte paths, '' to b'', and
// for integral division, etc.
With this, tbloom should pass with both python major versions (python
3 version tested was python 3.7.5).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Nov 2019 21:10:41 +0000 (15:10 -0600)]
Require LC_CTYPE to be iso-8859-1 for Python 3
Add a guard to compat (for now) that will crash if the LC_CYTPE isn't
ISO-8859-1 (aka Latin-1) when running under python3. We definitely
need this for handling argv safely (with at least python 3.7), and
since we're planning to require this arrangement anyway, let's start
now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 10 Sep 2019 06:56:02 +0000 (01:56 -0500)]
Prevent Python 3 from interfering with argv bytes
Python 3 insists on treating all arguments as Unicode, and and if the
bytes don't fit, it shoehorns them in anyway[1]. We need the raw,
original bytes in many cases (paths being the obvious example), and
Python claims they can be extracted via fsdecode.
But experimentation with 3.7 has demonstrated that while this is
necessary, it is not sufficient to handle all possible binary
arguments in at least a UTF-8 locale. The interpreter may crash at
startup with some (randomly generated) argument values:
Fatal Python error: _PyMainInterpreterConfig_Read: memory allocation failed
ValueError: character U+134bd2 is not in range [U+0000; U+10ffff]
Current thread 0x00007f2f0e1d8740 (most recent call first):
Traceback (most recent call last):
File "t/test-argv", line 28, in <module>
out = check_output(cmd)
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
To fix that, always set the encoding to ISO-8859-1 before launching
Python, which should hopefully (given that ISO-8859-1 is a single-byte
"pass through" encoding), prevent Python from interfering with the
arguments.
Add t/test-argv to perform randomized testing for clean argv
pass-through. At the moment, with Python 3.7.3, if I disable the code
before the python exec in cmd/bup-python, this test eventually
provokes the crash above (though not on every run).
[1] https://www.python.org/dev/peps/pep-0383/
Thanks to Aaron M. Ucko for pointing out LC_ALL had been overlooked in
an earlier version of this patch, and would have undone the
adjustments.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 22:46:38 +0000 (17:46 -0500)]
Simplify bup startup process
Now that we've moved bup to cmd/, we adjust bup-python at install
time, and we can rely on that to find the module and cmd dirs,
simplify the startup process. Let bup-python handle all the
PYTHONPATH settings, etc., and drop BUP_MAIN_EXE, relying instead on a
path.exe() that finds the binary via path.__file__.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 21:47:16 +0000 (16:47 -0500)]
Move bup to cmd/ and symlink ./bup to cmd/bup
Move main.py to cmd/bup and make ./bup a permanent symlink to it.
Make the installed bup executable a relative symlink to the
LIBDIR/cmd/bup file. Always use the location of the bup executable as
a way to locate the other subcommands and the lib and resource
directories, either when running from the source tree or from an
install tree.
This work finishes the switch to have all the commands to rely on the
bup-python wrapper so that we can establish some norms we'll need to
support Python 3. It should also allow us to simplify the main.py
startup process.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 22:23:42 +0000 (17:23 -0500)]
Install bup-python and rely on it everywhere
Add a cmd/python-cmd.sh template and use it to generate
cmd/bup-python. Install bup-python alongside all the other commands
and stop rewriting the command #! headers at install time, so that
they'll use it.
We're setting up all of the commands to rely on bup-python so that
we'll have a common place to establish some norms that we'll want in
order to support Python 3.
Though in fact, you can't actually run the python-based subcommands
independently because they don't have the right PYTHONPATH, etc.
That's something we'll also fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Oct 2019 17:49:14 +0000 (12:49 -0500)]
Move pwd grp functions to pwdgrp module; require binary fields
Move the pwd and grp related functions in helpers to a new more
focused pwdgrp module and adjust them (for py3) to require and provide
binary (bytes) values for the appropriate fields: name, passwd, etc.
These fields are arbitrary bytes with no encoding information on
platforms like Linux and *BSD, so must be handled that way.
Create our own Passwd and Group classes to carry the data, and make
them __slots__ based since they're quite a lot smaller, even than
namedtuples:
https://justmachinelearning.com/2019/07/29/python-consumes-a-lot-of-memory-or-how-to-reduce-the-size-of-objects/
See the DESIGN section on Python 3 strings for additional information.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Oct 2019 17:46:37 +0000 (12:46 -0500)]
username/userfullname: use struct names
Use the pwd names, not indexes. That's clearer and it'll support
upcoming py3 compatibility work. While we're at it, start making it
clear these are binary (bytes).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 30 Nov 2019 18:09:04 +0000 (12:09 -0600)]
_helpers: fix -Wsign-compare warnings
Python 3 adds -Wsign-compare to the compiler arguments. Address the
issues that this causes gcc to warn about. Disable the warnings for
the INTEGRAL_ASSIGNMENT_FITS macro since it's intentionally doing
something that the compiler can't tell is OK (assuming it is), add
some simple tests, and change the types and code in other places to
avoid the suspect comparisons. Currently clang doesn't produce the
warnings.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 19:16:10 +0000 (13:16 -0600)]
Makefile: append python 2 tests to common lists
Instead of maintaining two separate lists for python 2 and python 3
tests, just have one list of tests that work in both versions, and
then selectively append the tests that only work in python 2 when
appropriate.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 28 Nov 2019 08:01:02 +0000 (02:01 -0600)]
Remove inefficient (or will be) uses of buffer
On amd64 sys.getsizeof(b'') is a bit under 40 bytes across python 2/3,
while a buffer(b'') adds 64 bytes in python 2, and the memoryview(b'')
that replaces it in python 3 adds 200. So just copy the bytes unless
the added overhead is worth it.
And while we're here, fix a few more python 3 str/bytes compatability
issues, and remove some redundant offset arithemetic by leaning on
range() a bit more. (Not that it likely matters, but aside from being
simpler, this is apparently more efficient too, because it moves more
of the work to C).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 19:15:30 +0000 (13:15 -0600)]
hashsplit: join buffers more efficiently
Drop buffer_concat in favor of join_bytes, since we can use b''.join()
to join the two put() chunks in python 3, which should be notably more
efficient than copying the memoryview bytes before joining them.
In python2, also use join when we just have bytes, otherwise
accommodate the fact that you can't join bytes and buffers there, and
you can't append bytes to a buffer. The case we currently care about
should still end up being handled the same as befoere (in the first
clause), i.e. buf + bytes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 20:58:36 +0000 (14:58 -0600)]
fsck: report par2 parallel detection status
When attempting to detect whether or not par2 supports parallel
processing, don't suppress its stdout for now so that we'll see
anything unexpected (should be empty). When verbose, log what we
think we've detected along with any unexpected par2 stderr output.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Gernot Schulz [Fri, 22 Nov 2019 13:41:39 +0000 (14:41 +0100)]
fsck: Hide error when probing par2 capabilities
bup-fsck checks for the availability of par2's -t1 option by executing
it (for the reason see dc10c9172e1197c77c246ea75aae63388dcd641e). If
the option is unavailable, par2 returns an error ("Invalid option
specified: -t1") which is not muted by the -qq option.
To hide this harmless par2 error during bup-fsck invocations the
subprocess call needs to capture stderr.
Signed-off-by: Gernot Schulz <post@gernot-schulz.com>
[rlb@defaultvalue.org: expand abbreviated hash to full hash above] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Oct 2019 16:48:58 +0000 (11:48 -0500)]
vfs: rely on FakeLink's target in readlink
This could result in an error like:
File "/usr/local/lib/bup/bup/vfs.py", line 524, in _compute_item_size
return len(_readlink(repo, item.oid))
AttributeError: 'FakeLink' object has no attribute 'oid'
Thanks to Hartmut Krafft for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 9 Oct 2019 06:11:39 +0000 (01:11 -0500)]
Fix hashsplit for python 3; add compat.buffer
In order to get hashsplit working, stick with buffer()s in python 2.
We can't use a Py_buffer (via memoryview()) across both python
versions because python 2's zlib.compress() never learned how to
handle them, and we don't want to have to make an extra tobytes() copy
of all the data just to simplify the code.
Instead provide a python 3 only compat.buffer that's returns a
memoryview (slice when appropriate) in python 3.
Also noticed (and a bit unfortunate if accurate) that memoryview is
much larger than buffer, and for our bulk data, we have no need of any
additional sophistication:
$ python3
...
>>> import sys
>>> sys.getsizeof(buffer(''))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'buffer' is not defined
>>>
>>> sys.getsizeof(memoryview(b''))
192
Of course worth keeping in mind when deciding between using one or
just copying the relevant region.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 05:41:35 +0000 (00:41 -0500)]
Minimize use of preexec_fn
Python suggests avoiding preexec_fn, and in Python 3, changes to the
PATH no longer affect the subprocess invocation, so replace it with an
env setting where we can, and they've also added start_new_session to
provide setsid(), so we'll use that too when possible.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 10 Sep 2019 07:30:26 +0000 (02:30 -0500)]
Rename chain_ex to add_ex_ctx to reflect upstream terminology
In the raise[1] docs Python uses chaining to refer to "from" (cause)
linkage, not implicit exception/finally (context) linkage, so adjust
our function name to match. (Also preparation for compat code to
handle from linkage).
Rob Browning [Tue, 10 Sep 2019 06:24:27 +0000 (01:24 -0500)]
main: switch print_clean_line to bytes
This function filters arbitrary content, and only needs to assume that
it's ASCII-compatible (i.e. it only cares about \n and \r).
Explicitly use bytes to accommodate Python 3.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 10 Sep 2019 06:09:09 +0000 (01:09 -0500)]
bup_stat bup_lstat: always treat path as binary
Convert the path via "y" in Python 3, and rename the format alias to
cstr_argf to more accurately reflect what it means, i.e. a sequence of
non-null bytes. If "y" ends up being sufficient it's simpler (and
more efficient?) than a Py_buffer conversion, though it doesn't
support as many argument types.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 8 Sep 2019 19:39:36 +0000 (14:39 -0500)]
Have "make check" to do nothing for Python 3
In preparation for Python 3, don't run any tests for make check.
We'll enable individual tests once bup supports Python 3 well enough
for them to pass.
This shouldn't risk producing a false sense of security because you
still have to set BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true or bup
won't run at all, and if you've seen the warning produced when you
don't set it, you should be sufficiently well informed.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 11 Sep 2019 02:54:15 +0000 (21:54 -0500)]
test-get: rm pax_global_header from git archive output
git-archive now adds a pax_global_header, and while GNU tar suppresses
it, some versions of tar don't, so remove it explicitly to avoid
spurious test failures.
Thanks to Greg Troxel for for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 21:02:18 +0000 (16:02 -0500)]
cirrus-ci: specify make -j3 on all platforms
Based on past experience, this should help -- locally it has appeared
to help our tests even with a greater concurrency level than the cpu
count, and it looks like all the current test hosts have at least two
cores.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 18:18:17 +0000 (13:18 -0500)]
prep-for-macos-build: get brew rsync
It looks like macos may ship with a more limited rsync (or at least
one that doesn't produce the --itemize-changes output we expect, so
just install brew's.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 16:49:54 +0000 (11:49 -0500)]
fsck: only specify -t1 when par2 appears to support it
It looks like par2 may reject the "-t1" argument even when it's new
enough to support parallelism, so we can't rely on detecting the
version. Instead, before running the first real par2 command, test
-t1 in a sandbox to decide whether we can use it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 21 Aug 2019 06:23:30 +0000 (01:23 -0500)]
fsck: disable par2's internal concurrency
It looks like par2 added its own concurrency a bit back, and when
that's combined with bup's fsck -j parallelism, a simple
t/test-fsck.sh run completely swamps a machine here, launching what
looks like 32 threads total (for a 4/8 core machine), and fully
saturating the CPU.
The current test, which specifies -j99 actually ends up launching 4
par2 invocations, each processing a single file, and then each par2
appears to assume it has the entire machine to itself and launches 8
threads (one per hardware "core").
The resulting test takes 100s, but if we disable par2's parallelism
with -t1, putting bup's -j argument back in control of the overall
level of concurrency, the run time comes down to 4s.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 20 Aug 2019 01:34:11 +0000 (20:34 -0500)]
test-prune-older: disable background gc
The debian buildd's exposed a race between the default automatic
background gc and the explict gc in the test, resulting in:
git commit --date 1566025987 -qam 1566025987
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
git gc --aggressive
fatal: gc is already running on machine 'zandonai' pid 34323 (use --force if not)
Traceback (most recent call last):
File "t/test-prune-older", line 190, in <module>
save_utcs = create_older_random_saves(save_population, three_years_ago, now)
File "t/test-prune-older", line 74, in create_older_random_saves
exc(['git', 'gc', '--aggressive'])
File "t/test-prune-older", line 41, in exc
check_call(cmd, shell=shell)
File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'gc', '--aggressive']' returned non-zero exit status 128
Fix it by disabling gc.autoDetach in the git config for the test repo
-- something we may well need to do in other tests too.
Thanks to Robert Edmonds for reporting the problem and suggesting the
fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
If the watched process ends a push to the pipe without a newline at
the end, but with newlines in the middle, then sep_rx.split() will
return with multiple entries, the last of which will not end with a
newline and yet not be the empty string. This line prefix needs to be
stashed into the pending buffer, too.
This turns out to be exactly the same logic as if sep_rx.split had not
split the string, so eliminate one layer of conditionals.
This version incorporates feedback from Rob Browning to continue to
pass a list to extend().
Signed-off-by: Nathaniel Filardo <nwf20@cl.cam.ac.uk>
[rlb@defaultvalue.org: adjust commit summary and remove extra space in
"if split[0]" guard.] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Greg Troxel [Wed, 5 Jun 2019 20:55:32 +0000 (16:55 -0400)]
restore: create fifos with mkfifo, not mknod
I recently did a restore of a large bup backup, about 34G worth. All
worked well, including metadata, except that bup threw an exception on
restoring fifos (that I didn't need; they were in /var and were sockets
in use by daemons when the backup happened).
The problem was that mknod was being called for the fifo, and given only
two argumetns. mknod(2) on NetBSD says it takes three arguments.
mkfifo takes two. I am guessing that mknod in python calls mknod the OS
call, and on Linux somehow the third null argument works out somehow.
But it seems irregular to make a fifo with mknod.
I realize python is not POSIX, but mknod(2) requires three arguments:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/mknod.html
It would be nice to have a test of backing up and restoring a fifo; that
would have caught this.
The following patch makes my restore go smoothly.
Signed-off-by: Greg Troxel <gdt@lexort.com>
[rlb@defaultvalue.org: adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 13 Apr 2019 17:07:18 +0000 (12:07 -0500)]
rev_list: handle multiple results/ref from remote for custom formats
Previously a remote rev_list invocation would crash if a custom format
was provided and the remote produced multiple results for any input
ref. Change the client/server code to use a blank line to indicate
the end of the rev-list results. Of course, that means that the parse
function provided must be able to handle (consume) any blank lines
that its format string produces, which may preclude the use of some
format strings, but should be sufficient for now.
Adjust test-ls to (trivially) cover this case, and broaden the use of
the commit hash length tests in the code.
Thanks to Alex Roper for reporting the problem, providing an easy way
to reproduce it, and proposing a fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 12 Apr 2019 19:55:13 +0000 (14:55 -0500)]
Handle commit mergetags (at all)
Previously bup would just crash (i.e. bup ls) if it encountered a
commit with a mergetag header (apparently a new thing). For now,
adjust git.parse_commit to accept and ignore them as long as they only
appear as an optional, final header in the commit. That may or may
not turn out to be sufficient, and it does mean that for now we won't
be able to preserve mergetags (if we want to) whenever we rewrite
commits via bup gc, get, etc.
Rob Browning [Sun, 23 Mar 2014 17:41:06 +0000 (12:41 -0500)]
Add bup get; see the documentation for further information
WARNING: this is a new EXPERIMENTAL command that can (intentionally)
modify your data in destructive ways. Treat with caution.
Thanks to Karl Kiniger <karl.kiniger@med.ge.com> for helping track
down various bugs in earlier versions, and for noting that we might
want --verbose to be a little more effusive. And thanks to Patryck
Rouleau <prouleau72@gmail.com> for suggesting improvements to the
documentation.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 20:32:52 +0000 (15:32 -0500)]
repo: avoid cyclic dependency with is_remote method
The current VFS operations (like resolve()) require a repo object, but
we're about to add a VFS resolve() method to the repos. In and of
itself, that isn't necessarily a problem, but as an optimization, we
want the VFS resolve() to be able to detect when the repo it's been
given is a RemoteRepo and redirect the call to remote_repo.resolve().
Doing so pushes the one single resolve() call to the remote instead of
executing the resolve() locally with a lot of individual calls to the
remote_repo's other methods.
Adding is_remote() makes that possible without having to 'import repo'
in the VFS (repo already imports vfs).
Perhaps we'll rework it later, but this will do for now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 19:14:43 +0000 (14:14 -0500)]
Move vfs resolve() tests to tresolve.py
Move resolve() tests from tvfs to tresolve, and the common tree_dict()
test code to a new test.vfs module, in preparation for more extensive
resolve() testing.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Jan 2019 22:22:14 +0000 (16:22 -0600)]
test_resolve_loop: ensure exception is actually thrown
Make sure to resolve the correct path, and ensure the call never
returns. Previously when the path was wrong, and it *was* wrong, the
test would appear to succeed even though it wasn't actually testing
the intended ELOOP case.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 23:04:27 +0000 (18:04 -0500)]
metadata: always add/store/retrieve size for links and normal files
This simplifies cases where we need to transmit Metadata
objects (i.e. bup-get's repo.resolve()), and it means that for trees
created using this new v3 format, retrieving the sizes of chunked
files should be notably less expensive, since they'll be directly
available in the directory's .bupm file.
Without that, we have to seek around in the chunked tree to find the
last byte (cf. vfs._normal_or_chunked_file_size).
Only store the sizes for symlinks and regular files (which might be
chunked) until it's clear that other st_sizes are useful.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Feb 2018 07:32:54 +0000 (23:32 -0800)]
Replace lresolve with resolve(..., follow=False)
Although there's the NOFOLLOW precedent, this is really just to narrow
the API before we add it as a repo method, i.e. so we only have to
handle one function instead of two.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 9 Dec 2018 18:40:26 +0000 (12:40 -0600)]
vfs: change /save/latest back to a symlink to the latest save
The current, reworked vfs presents /somesave/latest as if it actually
is the latest commit. Change it back to a symlink to the latest save
to roughly match the previous behavior -- though now it's a link to
the save name, not to the (removed) /.commit/ subtree.
To restore the link, reintroduce the concept of a fake
symlink (i.e. one that has no corresponding blob in the repository).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 3 Dec 2018 18:58:03 +0000 (12:58 -0600)]
Don't return invalid data for offset reads (observed via fuse)
Fix a serious bug in the vfs that could cause it to return invalid
data for chunked file read()s that didn't start at the beginning of
the file. The issue was first observed via fuse, which makes sense
given that it streams a file in chunks that (currently) each come from
independent, increasing seek-offset FileReaders.
The previous dropwhile() invocation in the _tree_chunks generator,
used to skip past chunks that were completely before the offset, was
simple but wrong, and would skip too far. Replace it with
_skip_chunks_before_offset().
Add randomized tests of both simple streaming reads, and seek offset
reads, which catch the problem, cover additional cases, and should
prevent regressions.
Thanks to voldial for reporting the problem and providing an easy way
to reproduce it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log