Rob Browning [Sun, 6 May 2018 16:46:45 +0000 (11:46 -0500)]
vfs: cache resolve() calls to improve (fuse) performance
Include resolve() results in the vfs cache. This substantially
improves fuse "cat somefile" performance. (Observed a ~2x rate
improvement with a 500MB urandom file).
This appears to be due to the fact that fuse read(path, offset, len)
is called many times for the file, resulting in many corresponding,
redundant resolve(path) calls.
The previous fuse implementation, based on the previous vfs had its
own cache, but moving the caching to the vfs should be more generally
helpful.
Now bup fuse will again ignore repository changes that affect paths it
has already examined. This matches its behavior in the current stable
release (0.29.1).
Thanks to voldial for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Robert Evans [Sun, 29 Apr 2018 10:47:30 +0000 (06:47 -0400)]
Add bup split --noop <--blobs|--tree>
This prints the resulting id without storing in the repo.
Signed-off-by: Robert Evans <evansr@google.com>
[rlb@defaultvalue.org: remove trailing period from commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Patrick Rouleau [Mon, 23 Apr 2018 01:47:37 +0000 (21:47 -0400)]
fix t/root-status for CygWin
Signed-off-by: Patrick Rouleau <prouleau72@gmail.com> Tested-by: Patrick Rouleau <prouleau72@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 16:53:24 +0000 (11:53 -0500)]
is_superuser: test for group 544 or 0 on cygwin
This appears to be the appropriate way to check for admin status in
cygwin right now: https://cygwin.com/ml/cygwin/2015-02/msg00057.html
Thanks to at least Andrew Skretvedt, Ruvim Pinka, renpj, and Iar De
for reporting the problem, Ruvim Pinka, Paul Kronenwetter, and renpj
for proposing earlier solutions, and Ben Kelly and Johannes Berg for
helping test this approach.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 18:24:48 +0000 (13:24 -0500)]
buptest: base testing subproc funcs ex() and exo() on run()
Rework the subprocess functions, providing ex() and exo() as concise
test functions that print the commands they're executing. Base them
on a common, lower-level run() function.
Drop exc() since ex() and exo() both check the exit status by default.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:09:38 +0000 (23:09 -0500)]
Add "AND CONTRIBUTORS" to options.py license
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/JC9rm69oAQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 25 Mar 2018 21:35:58 -0400
Message-ID: <CAHqTa-2LB2mqscnzZmmixeTbR86BPh=FTk3UyGQKjPwaQPrZ3g@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
On Sun, Mar 25, 2018 at 1:48 PM, Rob Browning <rlb@defaultvalue.org> wrote:
> Avery Pennarun <apenwarr@gmail.com> writes:
>> On Sun, Mar 18, 2018 at 11:52 AM, Rob Browning <rlb@defaultvalue.org> wrote:
>>> Avery Pennarun <apenwarr@gmail.com> writes:
>>> So for the record, it sounds like you approve of changing the relevant
>>> bupsplit.h and bupsplit.c phrase to this?
>>>
>>> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
>>
>> Yes, that's fine.
>
> And perhaps you'd approve of making the same change to options.py as
> well?
Good idea. Let's do that :)
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 25 Mar 2018 17:19:53 +0000 (12:19 -0500)]
Add "AND CONTRIBUTORS" to bupsplit.h and bupsplit.c licenses
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/UyOe7VGuCQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 18 Mar 2018 18:35:15 -0400
Message-ID: <CAHqTa-1ghU6+Y0Y2pBOjbS=7CWKMytPvj-c1Z0aE3=PqpPi1OA@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
...
> So for the record, it sounds like you approve of changing the relevant
> bupsplit.h and bupsplit.c phrase to this?
>
> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
Yes, that's fine.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:29:17 +0000 (23:29 -0500)]
test-sparse-files: check sparse file size more carefully
Test the actual sparse file with du, not the parent restore
directory. Unnecessarily testing the whole tree broke the tests on a
docker system because the directories themselves were large.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 21:19:24 +0000 (15:19 -0600)]
main: don't set stdio to nonblocking
This was left in the newliner replacement patch, and I can't remember
a justification for it. Having it in there affects other processes,
so take it out.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 17:40:02 +0000 (11:40 -0600)]
Use absolute_import from the __future__ everywhere
Without this, among other things, we can end up with conflicts with
new upstream modules. For example, given lib/bup/io.py:
Traceback (most recent call last):
File "/home/rlb/src/bup/main-4/cmd/bup-index", line 10, in <module>
from bup import metadata, options, git, index, drecurse, hlinkdb
File "/home/rlb/src/bup/main-4/lib/bup/metadata.py", line 10, in <module>
from io import BytesIO
ImportError: cannot import name BytesIO
This switch also revealed a circular dependency between midx and git,
and several places where we weren't qualifying our bup imports
properly, i.e. "import git" rather than "from bup import git".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Jun 2016 20:49:14 +0000 (15:49 -0500)]
Clean subprocess output without newliner
Instead of launching a newliner subprocess from main.py to clean up the
output from other subprocesses (with respect to \r, shorter lines as
compared to the previous line, etc.), do something similar directly from
main.py via select, and then drop all of the attendant signal
forwarding, subprocess detaching via setsid(), etc. that the newliner
arrangement had accumulated.
Among other things, this should make bup much more responsive to
TERM/QUIT/etc. signals (i.e. C-c). Previously the
index/save/... subprocess might continue running even after bup
appeared to have quit.
To clean up the output, regularly check tty_width() and always write
"whole" space-padded lines, tracking stdout and stderr independently
(bup-newliner redirected both to stdout).
Rob Browning [Sun, 31 Dec 2017 18:23:01 +0000 (12:23 -0600)]
chain_ex: don't opportunistically add tracebacks to context_ex
Don't add a traceback to the context_ex while chaining. This could be
wrong, depending on the context, and in the common case, the traceback
more likely belonged on the result exception, not the context_ex.,
i.e.
except ... as ex:
raise chain_ex(ex, pending_ex)
Instead, just combine chain_ex with add_ex_tb when appropriate:
except ... as ex:
raise chain_ex(add_tb_ex(ex), pending_ex)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Without this "bup split -n NAME -d STAMP FILE" produces an error:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-split", line 67, in <module>
date = parse_date_or_fatal(opt.date, o.fatal)
NameError: name 'parse_date_or_fatal' is not defined
Signed-off-by: Fabian 'xx4h' Melters <xx4h@xx4h.de>
[rlb@defaultvalue.org: rebase on current master; adjust commit
message; fix commit author] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Nov 2017 19:24:41 +0000 (13:24 -0600)]
exc: support input arg; return SubprocInfo from exc and exo
Change exo to return a namedtuple so that you can refer to bits of the
result by name.
Base exc on exo so that it can support an input arg (even in python2),
and change both exc and exo to be a thinner wrapper around Popen so
that the arguments will behave roughly the same.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 13 Nov 2017 04:44:03 +0000 (22:44 -0600)]
vfs2: add trivial random eviction commit cache
Porting rm and gc to vfs2 has put us at the point where we need at
least some caching so that we're not invoking git an endless number of
times when trying to "bup rm" 1000 saves (i.e. via test-prune-older if
nothing else). Start with a very primitive, fixed-size commit cache
with random eviction.
It's not clear that the cache should always be enabled. While it may
help commands like rm, prune-older, fuse, web, ftp, etc., it may often
just be extra baggage for for commands like restore, ls, cat-file, ...
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 16:51:10 +0000 (10:51 -0600)]
vfs2: add tree_data_reader for use by cat-file
The cat-file command needs to be able to read arbitrary oids (chunked
or not) for --bupm, so add tree_data_reader for that purpose and
adjust fopen() to depend on it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 00:21:21 +0000 (18:21 -0600)]
ls: add --commit-hash and drop vfs nominal_oid
Report the tree hash for commits for ls -s, unless --commit-hash is
also specified, and add some initial tests. This brings -s in line
with 38a49e9eb930e8a47e9de5a9715fad659a774e8b.
Rob Browning [Sat, 9 Dec 2017 19:46:13 +0000 (13:46 -0600)]
vfs2._resolve_path: improve handling ENOTDIR, absolute paths, etc.
Handle and test various path_resolution(7) cases more carefully.
e.g. ensure a trailing '/' or '/.' forces resolution, ensure a
non-final path element that's not a directory produces ENOTDIR, ...
Add tests for the these, for bad symlink handling, and a for number of
other cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:55:23 +0000 (15:55 -0500)]
vfs2: report tree metadata for commits
Don't synthesize metadata for a commit (like /foo, /foo/2017-..., or
/.tag/some-commit) by reporting the author date as the mtime, instead
use the metadata for the underlying commit tree.
Among other things, this avoids an inconsistency between the metadata
for ".../commit" and ".../commit/.".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:41:33 +0000 (15:41 -0500)]
vfs2: add fill_in_metadata_if_dir and ensure_item_has_metadata
Add fill_in_metadata_if_dir() as a contents() helper since contents()
does not return metadata for directories (because it's expensive).
This supports:
(name, fill_in_metadata_if_dir(repo, item)
for name, item in tuple(contents(...)))
Add ensure_item_has_metadata() to combine fill_in_metadata_if_dir()
and augment_item_meta() for cases like "ls -l" where you want
every item to have all the metadata available, and to have a Metadata
instance (even if faked), not just a mode.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 25 Nov 2017 17:10:03 +0000 (11:10 -0600)]
vfs2._resolve_path: require parent to be a path sequence
Require an actual path sequence for the parent since it provides the
context for the resolution of the path (with respect to relative
paths, symlinks containing .., etc.).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Oct 2017 19:40:15 +0000 (14:40 -0500)]
vfs2: suffix duplicate save names (e.g. 1970-01-01-214640-07)
From now on, append zero-padded integers to the names of saves that
have the same commit time in order to avoid duplicates. The sequence
number currently represents the save's reversed position in default
rev-list order, so that given:
Rob Browning [Sat, 11 Nov 2017 17:53:56 +0000 (11:53 -0600)]
test-rm: fix regexes to handle newer *and* older git
To accommodate optional lines, we need to use a construct like this:
foo(
bar)?
instead of this:
foo
(bar)?
The problem became apparent when people tested against older git after
the introduction of the patch to work with newer git: 292361d86d1cf0cc555681ae43371d66c8ebb366
Thanks to Greg Troxel and Basil Mohamed Gohar for reporting the
problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log