Rob Browning [Mon, 3 Dec 2018 18:58:03 +0000 (12:58 -0600)]
Don't return invalid data for offset reads (observed via fuse)
Fix a serious bug in the vfs that could cause it to return invalid
data for chunked file read()s that didn't start at the beginning of
the file. The issue was first observed via fuse, which makes sense
given that it streams a file in chunks that (currently) each come from
independent, increasing seek-offset FileReaders.
The previous dropwhile() invocation in the _tree_chunks generator,
used to skip past chunks that were completely before the offset, was
simple but wrong, and would skip too far. Replace it with
_skip_chunks_before_offset().
Add randomized tests of both simple streaming reads, and seek offset
reads, which catch the problem, cover additional cases, and should
prevent regressions.
Thanks to voldial for reporting the problem and providing an easy way
to reproduce it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 17:35:11 +0000 (12:35 -0500)]
update-doc-branches: add command to update man and html
Create a new command to update the man and html branches, and move the
related code there from the Makefile.
Update the branches based on the current (clean) tree, rather than
consulting the git origin, and rely on ls-files rather than globbing
so that the file lists will always be correct -- we'll immediately
notice deletions, avoid picking up stray files in the directory, etc.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 30 Sep 2018 20:29:42 +0000 (15:29 -0500)]
Makefile: fix find -printf issue on FreeBSD
Apparently the use of -printf was causing the error: "printf: missing
format character" with FreeBSD 11.1-RELEASE. Change the helpers lib
count to rely on -print0 and tr, which should be more portable, but
still be undisturbed by unusual paths.
Thanks to Curtis Dunham for reporting the problem and proposing an
alternate solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 7 Jul 2018 17:20:47 +0000 (12:20 -0500)]
vfs: flatten resolution cache key
This will require more storage if there are a lot of lookups with the
same parent and differing paths, but otherwise, without more
intentional structure sharing among paths, this should be better, and
we can always revisit the arrangement later.
Serializing the parent path segments should also make sure the same
parent (semantically-speaking) contributes the same hash to the key.
Previously, Metadata objects could prevent that, given their trivial,
pointer-based hashes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 2 Sep 2018 17:29:01 +0000 (12:29 -0500)]
Refuse to run by default if python version is not 2
Exit with an error if the python major version isn't 2, since we're
working on support for py3, and we'll probably reach a point before
we're finished where bup doesn't immediately crash with py3, but might
well do very bad things to the repository.
Allow overriding the check by setting
BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true so people can still test py3
if they like.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 09:57:18 +0000 (19:57 +1000)]
tindex: add 0o to current octal literal
Needed for python3.
This preserves current behaviour, but since I get no test failures if I
just chop of the leading 0 and make it a normal int there's a good
chance there's underlying breakage.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: adjust commit message summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 08:25:25 +0000 (18:25 +1000)]
Replace remaining print statements with print function
Also add the appropriate __future__ import statement.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: memtest-cmd.py: remove a few preexisting
trailing spaces that were carried over; adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Jul 2018 17:01:55 +0000 (12:01 -0500)]
vfs: include unique repo id in resolution cache key
...since resolve() currently requires a full parent path and the root
refs are only applicable to a particular repository.
Use differing integers to identify repositories that may be
independent (with respect to refs, tags, etc.), and use (typically
small) integers rather than the repo path/address so that they'll be
short if we want to embed them directly in cache keys later.
Use realpath() for local repositories in order to detect when the same
repository is reachable by multiple paths. (Something similar could
eventually be done for remotes.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 6 May 2018 16:46:45 +0000 (11:46 -0500)]
vfs: cache resolve() calls to improve (fuse) performance
Include resolve() results in the vfs cache. This substantially
improves fuse "cat somefile" performance. (Observed a ~2x rate
improvement with a 500MB urandom file).
This appears to be due to the fact that fuse read(path, offset, len)
is called many times for the file, resulting in many corresponding,
redundant resolve(path) calls.
The previous fuse implementation, based on the previous vfs had its
own cache, but moving the caching to the vfs should be more generally
helpful.
Now bup fuse will again ignore repository changes that affect paths it
has already examined. This matches its behavior in the current stable
release (0.29.1).
Thanks to voldial for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Robert Evans [Sun, 29 Apr 2018 10:47:30 +0000 (06:47 -0400)]
Add bup split --noop <--blobs|--tree>
This prints the resulting id without storing in the repo.
Signed-off-by: Robert Evans <evansr@google.com>
[rlb@defaultvalue.org: remove trailing period from commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Patrick Rouleau [Mon, 23 Apr 2018 01:47:37 +0000 (21:47 -0400)]
fix t/root-status for CygWin
Signed-off-by: Patrick Rouleau <prouleau72@gmail.com> Tested-by: Patrick Rouleau <prouleau72@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 16:53:24 +0000 (11:53 -0500)]
is_superuser: test for group 544 or 0 on cygwin
This appears to be the appropriate way to check for admin status in
cygwin right now: https://cygwin.com/ml/cygwin/2015-02/msg00057.html
Thanks to at least Andrew Skretvedt, Ruvim Pinka, renpj, and Iar De
for reporting the problem, Ruvim Pinka, Paul Kronenwetter, and renpj
for proposing earlier solutions, and Ben Kelly and Johannes Berg for
helping test this approach.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 18:24:48 +0000 (13:24 -0500)]
buptest: base testing subproc funcs ex() and exo() on run()
Rework the subprocess functions, providing ex() and exo() as concise
test functions that print the commands they're executing. Base them
on a common, lower-level run() function.
Drop exc() since ex() and exo() both check the exit status by default.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:09:38 +0000 (23:09 -0500)]
Add "AND CONTRIBUTORS" to options.py license
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/JC9rm69oAQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 25 Mar 2018 21:35:58 -0400
Message-ID: <CAHqTa-2LB2mqscnzZmmixeTbR86BPh=FTk3UyGQKjPwaQPrZ3g@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
On Sun, Mar 25, 2018 at 1:48 PM, Rob Browning <rlb@defaultvalue.org> wrote:
> Avery Pennarun <apenwarr@gmail.com> writes:
>> On Sun, Mar 18, 2018 at 11:52 AM, Rob Browning <rlb@defaultvalue.org> wrote:
>>> Avery Pennarun <apenwarr@gmail.com> writes:
>>> So for the record, it sounds like you approve of changing the relevant
>>> bupsplit.h and bupsplit.c phrase to this?
>>>
>>> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
>>
>> Yes, that's fine.
>
> And perhaps you'd approve of making the same change to options.py as
> well?
Good idea. Let's do that :)
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 25 Mar 2018 17:19:53 +0000 (12:19 -0500)]
Add "AND CONTRIBUTORS" to bupsplit.h and bupsplit.c licenses
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/UyOe7VGuCQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 18 Mar 2018 18:35:15 -0400
Message-ID: <CAHqTa-1ghU6+Y0Y2pBOjbS=7CWKMytPvj-c1Z0aE3=PqpPi1OA@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
...
> So for the record, it sounds like you approve of changing the relevant
> bupsplit.h and bupsplit.c phrase to this?
>
> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
Yes, that's fine.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:29:17 +0000 (23:29 -0500)]
test-sparse-files: check sparse file size more carefully
Test the actual sparse file with du, not the parent restore
directory. Unnecessarily testing the whole tree broke the tests on a
docker system because the directories themselves were large.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 21:19:24 +0000 (15:19 -0600)]
main: don't set stdio to nonblocking
This was left in the newliner replacement patch, and I can't remember
a justification for it. Having it in there affects other processes,
so take it out.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 17:40:02 +0000 (11:40 -0600)]
Use absolute_import from the __future__ everywhere
Without this, among other things, we can end up with conflicts with
new upstream modules. For example, given lib/bup/io.py:
Traceback (most recent call last):
File "/home/rlb/src/bup/main-4/cmd/bup-index", line 10, in <module>
from bup import metadata, options, git, index, drecurse, hlinkdb
File "/home/rlb/src/bup/main-4/lib/bup/metadata.py", line 10, in <module>
from io import BytesIO
ImportError: cannot import name BytesIO
This switch also revealed a circular dependency between midx and git,
and several places where we weren't qualifying our bup imports
properly, i.e. "import git" rather than "from bup import git".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Jun 2016 20:49:14 +0000 (15:49 -0500)]
Clean subprocess output without newliner
Instead of launching a newliner subprocess from main.py to clean up the
output from other subprocesses (with respect to \r, shorter lines as
compared to the previous line, etc.), do something similar directly from
main.py via select, and then drop all of the attendant signal
forwarding, subprocess detaching via setsid(), etc. that the newliner
arrangement had accumulated.
Among other things, this should make bup much more responsive to
TERM/QUIT/etc. signals (i.e. C-c). Previously the
index/save/... subprocess might continue running even after bup
appeared to have quit.
To clean up the output, regularly check tty_width() and always write
"whole" space-padded lines, tracking stdout and stderr independently
(bup-newliner redirected both to stdout).
Rob Browning [Sun, 31 Dec 2017 18:23:01 +0000 (12:23 -0600)]
chain_ex: don't opportunistically add tracebacks to context_ex
Don't add a traceback to the context_ex while chaining. This could be
wrong, depending on the context, and in the common case, the traceback
more likely belonged on the result exception, not the context_ex.,
i.e.
except ... as ex:
raise chain_ex(ex, pending_ex)
Instead, just combine chain_ex with add_ex_tb when appropriate:
except ... as ex:
raise chain_ex(add_tb_ex(ex), pending_ex)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Without this "bup split -n NAME -d STAMP FILE" produces an error:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-split", line 67, in <module>
date = parse_date_or_fatal(opt.date, o.fatal)
NameError: name 'parse_date_or_fatal' is not defined
Signed-off-by: Fabian 'xx4h' Melters <xx4h@xx4h.de>
[rlb@defaultvalue.org: rebase on current master; adjust commit
message; fix commit author] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Nov 2017 19:24:41 +0000 (13:24 -0600)]
exc: support input arg; return SubprocInfo from exc and exo
Change exo to return a namedtuple so that you can refer to bits of the
result by name.
Base exc on exo so that it can support an input arg (even in python2),
and change both exc and exo to be a thinner wrapper around Popen so
that the arguments will behave roughly the same.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 13 Nov 2017 04:44:03 +0000 (22:44 -0600)]
vfs2: add trivial random eviction commit cache
Porting rm and gc to vfs2 has put us at the point where we need at
least some caching so that we're not invoking git an endless number of
times when trying to "bup rm" 1000 saves (i.e. via test-prune-older if
nothing else). Start with a very primitive, fixed-size commit cache
with random eviction.
It's not clear that the cache should always be enabled. While it may
help commands like rm, prune-older, fuse, web, ftp, etc., it may often
just be extra baggage for for commands like restore, ls, cat-file, ...
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 16:51:10 +0000 (10:51 -0600)]
vfs2: add tree_data_reader for use by cat-file
The cat-file command needs to be able to read arbitrary oids (chunked
or not) for --bupm, so add tree_data_reader for that purpose and
adjust fopen() to depend on it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>