Rob Browning [Sat, 7 Jul 2018 17:20:47 +0000 (12:20 -0500)]
vfs: flatten resolution cache key
This will require more storage if there are a lot of lookups with the
same parent and differing paths, but otherwise, without more
intentional structure sharing among paths, this should be better, and
we can always revisit the arrangement later.
Serializing the parent path segments should also make sure the same
parent (semantically-speaking) contributes the same hash to the key.
Previously, Metadata objects could prevent that, given their trivial,
pointer-based hashes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 2 Sep 2018 17:29:01 +0000 (12:29 -0500)]
Refuse to run by default if python version is not 2
Exit with an error if the python major version isn't 2, since we're
working on support for py3, and we'll probably reach a point before
we're finished where bup doesn't immediately crash with py3, but might
well do very bad things to the repository.
Allow overriding the check by setting
BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true so people can still test py3
if they like.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 09:57:18 +0000 (19:57 +1000)]
tindex: add 0o to current octal literal
Needed for python3.
This preserves current behaviour, but since I get no test failures if I
just chop of the leading 0 and make it a normal int there's a good
chance there's underlying breakage.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: adjust commit message summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 08:25:25 +0000 (18:25 +1000)]
Replace remaining print statements with print function
Also add the appropriate __future__ import statement.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: memtest-cmd.py: remove a few preexisting
trailing spaces that were carried over; adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Jul 2018 17:01:55 +0000 (12:01 -0500)]
vfs: include unique repo id in resolution cache key
...since resolve() currently requires a full parent path and the root
refs are only applicable to a particular repository.
Use differing integers to identify repositories that may be
independent (with respect to refs, tags, etc.), and use (typically
small) integers rather than the repo path/address so that they'll be
short if we want to embed them directly in cache keys later.
Use realpath() for local repositories in order to detect when the same
repository is reachable by multiple paths. (Something similar could
eventually be done for remotes.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 6 May 2018 16:46:45 +0000 (11:46 -0500)]
vfs: cache resolve() calls to improve (fuse) performance
Include resolve() results in the vfs cache. This substantially
improves fuse "cat somefile" performance. (Observed a ~2x rate
improvement with a 500MB urandom file).
This appears to be due to the fact that fuse read(path, offset, len)
is called many times for the file, resulting in many corresponding,
redundant resolve(path) calls.
The previous fuse implementation, based on the previous vfs had its
own cache, but moving the caching to the vfs should be more generally
helpful.
Now bup fuse will again ignore repository changes that affect paths it
has already examined. This matches its behavior in the current stable
release (0.29.1).
Thanks to voldial for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Robert Evans [Sun, 29 Apr 2018 10:47:30 +0000 (06:47 -0400)]
Add bup split --noop <--blobs|--tree>
This prints the resulting id without storing in the repo.
Signed-off-by: Robert Evans <evansr@google.com>
[rlb@defaultvalue.org: remove trailing period from commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Patrick Rouleau [Mon, 23 Apr 2018 01:47:37 +0000 (21:47 -0400)]
fix t/root-status for CygWin
Signed-off-by: Patrick Rouleau <prouleau72@gmail.com> Tested-by: Patrick Rouleau <prouleau72@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 16:53:24 +0000 (11:53 -0500)]
is_superuser: test for group 544 or 0 on cygwin
This appears to be the appropriate way to check for admin status in
cygwin right now: https://cygwin.com/ml/cygwin/2015-02/msg00057.html
Thanks to at least Andrew Skretvedt, Ruvim Pinka, renpj, and Iar De
for reporting the problem, Ruvim Pinka, Paul Kronenwetter, and renpj
for proposing earlier solutions, and Ben Kelly and Johannes Berg for
helping test this approach.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 18:24:48 +0000 (13:24 -0500)]
buptest: base testing subproc funcs ex() and exo() on run()
Rework the subprocess functions, providing ex() and exo() as concise
test functions that print the commands they're executing. Base them
on a common, lower-level run() function.
Drop exc() since ex() and exo() both check the exit status by default.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:09:38 +0000 (23:09 -0500)]
Add "AND CONTRIBUTORS" to options.py license
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/JC9rm69oAQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 25 Mar 2018 21:35:58 -0400
Message-ID: <CAHqTa-2LB2mqscnzZmmixeTbR86BPh=FTk3UyGQKjPwaQPrZ3g@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
On Sun, Mar 25, 2018 at 1:48 PM, Rob Browning <rlb@defaultvalue.org> wrote:
> Avery Pennarun <apenwarr@gmail.com> writes:
>> On Sun, Mar 18, 2018 at 11:52 AM, Rob Browning <rlb@defaultvalue.org> wrote:
>>> Avery Pennarun <apenwarr@gmail.com> writes:
>>> So for the record, it sounds like you approve of changing the relevant
>>> bupsplit.h and bupsplit.c phrase to this?
>>>
>>> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
>>
>> Yes, that's fine.
>
> And perhaps you'd approve of making the same change to options.py as
> well?
Good idea. Let's do that :)
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 25 Mar 2018 17:19:53 +0000 (12:19 -0500)]
Add "AND CONTRIBUTORS" to bupsplit.h and bupsplit.c licenses
From https://groups.google.com/d/msg/bup-list/8lcxXuXilTE/UyOe7VGuCQAJ
From: Avery Pennarun <apenwarr@gmail.com>
Date: Sun, 18 Mar 2018 18:35:15 -0400
Message-ID: <CAHqTa-1ghU6+Y0Y2pBOjbS=7CWKMytPvj-c1Z0aE3=PqpPi1OA@mail.gmail.com>
Subject: Re: bupsplit.c copyright and patching
To: Rob Browning <rlb@defaultvalue.org> Cc: Robert Evans <evansr@google.com>, bup-list <bup-list@googlegroups.com>
...
> So for the record, it sounds like you approve of changing the relevant
> bupsplit.h and bupsplit.c phrase to this?
>
> THIS SOFTWARE IS PROVIDED BY AVERY PENNARUN AND CONTRIBUTORS``AS IS'' AND ANY
Yes, that's fine.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Mar 2018 04:29:17 +0000 (23:29 -0500)]
test-sparse-files: check sparse file size more carefully
Test the actual sparse file with du, not the parent restore
directory. Unnecessarily testing the whole tree broke the tests on a
docker system because the directories themselves were large.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 21:19:24 +0000 (15:19 -0600)]
main: don't set stdio to nonblocking
This was left in the newliner replacement patch, and I can't remember
a justification for it. Having it in there affects other processes,
so take it out.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 17:40:02 +0000 (11:40 -0600)]
Use absolute_import from the __future__ everywhere
Without this, among other things, we can end up with conflicts with
new upstream modules. For example, given lib/bup/io.py:
Traceback (most recent call last):
File "/home/rlb/src/bup/main-4/cmd/bup-index", line 10, in <module>
from bup import metadata, options, git, index, drecurse, hlinkdb
File "/home/rlb/src/bup/main-4/lib/bup/metadata.py", line 10, in <module>
from io import BytesIO
ImportError: cannot import name BytesIO
This switch also revealed a circular dependency between midx and git,
and several places where we weren't qualifying our bup imports
properly, i.e. "import git" rather than "from bup import git".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Jun 2016 20:49:14 +0000 (15:49 -0500)]
Clean subprocess output without newliner
Instead of launching a newliner subprocess from main.py to clean up the
output from other subprocesses (with respect to \r, shorter lines as
compared to the previous line, etc.), do something similar directly from
main.py via select, and then drop all of the attendant signal
forwarding, subprocess detaching via setsid(), etc. that the newliner
arrangement had accumulated.
Among other things, this should make bup much more responsive to
TERM/QUIT/etc. signals (i.e. C-c). Previously the
index/save/... subprocess might continue running even after bup
appeared to have quit.
To clean up the output, regularly check tty_width() and always write
"whole" space-padded lines, tracking stdout and stderr independently
(bup-newliner redirected both to stdout).
Rob Browning [Sun, 31 Dec 2017 18:23:01 +0000 (12:23 -0600)]
chain_ex: don't opportunistically add tracebacks to context_ex
Don't add a traceback to the context_ex while chaining. This could be
wrong, depending on the context, and in the common case, the traceback
more likely belonged on the result exception, not the context_ex.,
i.e.
except ... as ex:
raise chain_ex(ex, pending_ex)
Instead, just combine chain_ex with add_ex_tb when appropriate:
except ... as ex:
raise chain_ex(add_tb_ex(ex), pending_ex)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Without this "bup split -n NAME -d STAMP FILE" produces an error:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-split", line 67, in <module>
date = parse_date_or_fatal(opt.date, o.fatal)
NameError: name 'parse_date_or_fatal' is not defined
Signed-off-by: Fabian 'xx4h' Melters <xx4h@xx4h.de>
[rlb@defaultvalue.org: rebase on current master; adjust commit
message; fix commit author] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Nov 2017 19:24:41 +0000 (13:24 -0600)]
exc: support input arg; return SubprocInfo from exc and exo
Change exo to return a namedtuple so that you can refer to bits of the
result by name.
Base exc on exo so that it can support an input arg (even in python2),
and change both exc and exo to be a thinner wrapper around Popen so
that the arguments will behave roughly the same.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 13 Nov 2017 04:44:03 +0000 (22:44 -0600)]
vfs2: add trivial random eviction commit cache
Porting rm and gc to vfs2 has put us at the point where we need at
least some caching so that we're not invoking git an endless number of
times when trying to "bup rm" 1000 saves (i.e. via test-prune-older if
nothing else). Start with a very primitive, fixed-size commit cache
with random eviction.
It's not clear that the cache should always be enabled. While it may
help commands like rm, prune-older, fuse, web, ftp, etc., it may often
just be extra baggage for for commands like restore, ls, cat-file, ...
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 16:51:10 +0000 (10:51 -0600)]
vfs2: add tree_data_reader for use by cat-file
The cat-file command needs to be able to read arbitrary oids (chunked
or not) for --bupm, so add tree_data_reader for that purpose and
adjust fopen() to depend on it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 00:21:21 +0000 (18:21 -0600)]
ls: add --commit-hash and drop vfs nominal_oid
Report the tree hash for commits for ls -s, unless --commit-hash is
also specified, and add some initial tests. This brings -s in line
with 38a49e9eb930e8a47e9de5a9715fad659a774e8b.
Rob Browning [Sat, 9 Dec 2017 19:46:13 +0000 (13:46 -0600)]
vfs2._resolve_path: improve handling ENOTDIR, absolute paths, etc.
Handle and test various path_resolution(7) cases more carefully.
e.g. ensure a trailing '/' or '/.' forces resolution, ensure a
non-final path element that's not a directory produces ENOTDIR, ...
Add tests for the these, for bad symlink handling, and a for number of
other cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:55:23 +0000 (15:55 -0500)]
vfs2: report tree metadata for commits
Don't synthesize metadata for a commit (like /foo, /foo/2017-..., or
/.tag/some-commit) by reporting the author date as the mtime, instead
use the metadata for the underlying commit tree.
Among other things, this avoids an inconsistency between the metadata
for ".../commit" and ".../commit/.".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:41:33 +0000 (15:41 -0500)]
vfs2: add fill_in_metadata_if_dir and ensure_item_has_metadata
Add fill_in_metadata_if_dir() as a contents() helper since contents()
does not return metadata for directories (because it's expensive).
This supports:
(name, fill_in_metadata_if_dir(repo, item)
for name, item in tuple(contents(...)))
Add ensure_item_has_metadata() to combine fill_in_metadata_if_dir()
and augment_item_meta() for cases like "ls -l" where you want
every item to have all the metadata available, and to have a Metadata
instance (even if faked), not just a mode.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log