Rob Browning [Sun, 26 Nov 2017 19:24:41 +0000 (13:24 -0600)]
exc: support input arg; return SubprocInfo from exc and exo
Change exo to return a namedtuple so that you can refer to bits of the
result by name.
Base exc on exo so that it can support an input arg (even in python2),
and change both exc and exo to be a thinner wrapper around Popen so
that the arguments will behave roughly the same.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 13 Nov 2017 04:44:03 +0000 (22:44 -0600)]
vfs2: add trivial random eviction commit cache
Porting rm and gc to vfs2 has put us at the point where we need at
least some caching so that we're not invoking git an endless number of
times when trying to "bup rm" 1000 saves (i.e. via test-prune-older if
nothing else). Start with a very primitive, fixed-size commit cache
with random eviction.
It's not clear that the cache should always be enabled. While it may
help commands like rm, prune-older, fuse, web, ftp, etc., it may often
just be extra baggage for for commands like restore, ls, cat-file, ...
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 16:51:10 +0000 (10:51 -0600)]
vfs2: add tree_data_reader for use by cat-file
The cat-file command needs to be able to read arbitrary oids (chunked
or not) for --bupm, so add tree_data_reader for that purpose and
adjust fopen() to depend on it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 00:21:21 +0000 (18:21 -0600)]
ls: add --commit-hash and drop vfs nominal_oid
Report the tree hash for commits for ls -s, unless --commit-hash is
also specified, and add some initial tests. This brings -s in line
with 38a49e9eb930e8a47e9de5a9715fad659a774e8b.
Rob Browning [Sat, 9 Dec 2017 19:46:13 +0000 (13:46 -0600)]
vfs2._resolve_path: improve handling ENOTDIR, absolute paths, etc.
Handle and test various path_resolution(7) cases more carefully.
e.g. ensure a trailing '/' or '/.' forces resolution, ensure a
non-final path element that's not a directory produces ENOTDIR, ...
Add tests for the these, for bad symlink handling, and a for number of
other cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:55:23 +0000 (15:55 -0500)]
vfs2: report tree metadata for commits
Don't synthesize metadata for a commit (like /foo, /foo/2017-..., or
/.tag/some-commit) by reporting the author date as the mtime, instead
use the metadata for the underlying commit tree.
Among other things, this avoids an inconsistency between the metadata
for ".../commit" and ".../commit/.".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:41:33 +0000 (15:41 -0500)]
vfs2: add fill_in_metadata_if_dir and ensure_item_has_metadata
Add fill_in_metadata_if_dir() as a contents() helper since contents()
does not return metadata for directories (because it's expensive).
This supports:
(name, fill_in_metadata_if_dir(repo, item)
for name, item in tuple(contents(...)))
Add ensure_item_has_metadata() to combine fill_in_metadata_if_dir()
and augment_item_meta() for cases like "ls -l" where you want
every item to have all the metadata available, and to have a Metadata
instance (even if faked), not just a mode.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 25 Nov 2017 17:10:03 +0000 (11:10 -0600)]
vfs2._resolve_path: require parent to be a path sequence
Require an actual path sequence for the parent since it provides the
context for the resolution of the path (with respect to relative
paths, symlinks containing .., etc.).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Oct 2017 19:40:15 +0000 (14:40 -0500)]
vfs2: suffix duplicate save names (e.g. 1970-01-01-214640-07)
From now on, append zero-padded integers to the names of saves that
have the same commit time in order to avoid duplicates. The sequence
number currently represents the save's reversed position in default
rev-list order, so that given:
Rob Browning [Sat, 11 Nov 2017 17:53:56 +0000 (11:53 -0600)]
test-rm: fix regexes to handle newer *and* older git
To accommodate optional lines, we need to use a construct like this:
foo(
bar)?
instead of this:
foo
(bar)?
The problem became apparent when people tested against older git after
the introduction of the patch to work with newer git: 292361d86d1cf0cc555681ae43371d66c8ebb366
Thanks to Greg Troxel and Basil Mohamed Gohar for reporting the
problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 21 Oct 2017 16:27:15 +0000 (11:27 -0500)]
test-rm: accommodate packed-refs creation by newer git
It looks like newer versions of git (as of at least 2.150 rc1) are
automatically creating packed-refs. Adjust test-rm.sh to accommodate
that possibility.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Sep 2017 17:08:44 +0000 (12:08 -0500)]
client: ensure server provides requested command
Gather the list of available commands from the server during client
initialization and use that to throw a suitable ClientError whenever
the server doesn't support a requested command.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
This will be more heavily used later and exceptions are notably more
expensive. Further, a missing object isn't necessarily an error.
It's perfectly reasonable to attempt to check an object's existence
via get().
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Sep 2017 20:08:33 +0000 (15:08 -0500)]
Use next(it), not it.next() and drop the helpers fallback
Change all it.next() invocations to next(it) for compatibility with
newer versions of python that do not support the next() method. Drop
the next() fallback helper since we depend on python 2.6 now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 11 Jul 2017 20:53:33 +0000 (15:53 -0500)]
test-meta: don't try to chattr +T a file
...since 'T' only works for directories, and newer kernels actually
reject the attempt (as of at least 4.12, and maybe 4.10). Thanks to
Mateusz Pavlic for reporting the problem and verifying the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 18 Jun 2017 20:40:41 +0000 (15:40 -0500)]
Support catpipe get(...,size=True); require git >= 1.5.6
Add a new size argument to get() that requests the object size in
addition to the type. To support this, require git 1.5.6 (circa 2008)
or newer so that we'll have cat-file --batch.
Remove the _slow_get() fallback since it's no longer needed.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 7 Apr 2017 00:32:50 +0000 (19:32 -0500)]
Support exception chaining and tracebacks
Python 3 has added support for exception chaining
https://www.python.org/dev/peps/pep-3134/
Which makes it possible to avoid losing information when an exception
is thrown from within an exception handler:
try:
...
raise lp0_on_fire()
except Exception as ex:
# Imagine the disk is also full and close() throws too
some_output_file.close()
In this situation, you'll never find out the printer's on fire. With
chaining, the first exception will be attached to the second as its
__context__. (Note that "finally" blocks suffer from the same issue.)
The PEP also describes adding a __traceback__ attribute to exceptions
so that they're more self-contained.
Python 3 handles all of this automatically, and includes any chained
exceptions in its tracebacks. For this:
$ python3 lib/bup/compat.py
Traceback (most recent call last):
File "lib/bup/compat.py", line 74, in outer
inner()
File "lib/bup/compat.py", line 70, in inner
raise Exception('first')
Exception: first
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "lib/bup/compat.py", line 78, in <module>
wrap_main(outer)
File "lib/bup/compat.py", line 50, in wrap_main
sys.exit(main())
File "lib/bup/compat.py", line 76, in outer
raise chain_ex(Exception('second'), ex)
Exception: second
Add a compat.py supporting something similar for Python 2, and use it
in main.py.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Ben Kelly [Tue, 14 Mar 2017 16:29:04 +0000 (12:29 -0400)]
prune-older: keep most recent save in each period group
Keep the most recent save in each period group (day, week, month, ...)
rather than the oldest.
Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Ben Kelly [Tue, 21 Jan 2014 15:42:15 +0000 (10:42 -0500)]
bloom: end progress message with \r, not \n
This avoids leaving spurious output lines behind at exit.
Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Saveguard against deleting new pack-file (f.e. with threshold=0)
Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
[rlb@defaultvalue.org: wrap comment line in test-gc.sh; adjust comment
whitespace in gc.py] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 25 Dec 2016 17:44:56 +0000 (11:44 -0600)]
index: only collect metadata for stale paths
Stop collecting all of the metadata for every path, even if the path
hasn't changed since the last save. Further, rework the code to
short-circuit some other unnecessary work.
To support this, split Entry.from_stat() into two parts, stale(), which
tests to see if an entry has "materially" changed, and
update_from_stat() which updates the entry to match the stat information
provided.
This should substantially decrease the indexing cost for paths that
haven't changed since the last save.
While we're here, rename hashgen to fake_hash so it's clearer that's its
only purpose.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 17 Dec 2016 20:40:02 +0000 (14:40 -0600)]
git.cp(): don't repeatedly recompute default repo
Before, bup would end up calling repo() for every invocation of cp().
Given that constructions like cp().get(id) are common, this could cause
a lot of unecessary calls, given that we already have git.repodir, which
will have the relevant value once initialized.
Discovered after noticing "bup restore" was hammering
"$BUP_DIR"/.git (which didn't exist).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 10 Dec 2016 17:57:05 +0000 (11:57 -0600)]
gc: restart catpipe after each new pack
Without this, the space taken by the stale packfiles that gc removes
won't be available to the filesystem until the entire gc run ends (or
the catpipe is otherwise reset).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 15 Dec 2016 18:45:19 +0000 (12:45 -0600)]
test-prune-older: avoid duplicate save times
When creating the list of random save times make sure to avoid any
duplicates since the timestamp is also the trivial commit content.
Otherwise the git commit for the second save dies with "nothing to
commit, working tree clean".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 30 Oct 2016 18:31:43 +0000 (13:31 -0500)]
Add bup prune-older command
prune-older removes (permanently deletes) all saves except those
preserved by various temporal keep arguments. It is equivalent to a
suitable "bup rm" invocation followed by "bup gc".
For example, this invocation keeps all saves on the BRANCH for the past
month, and any older monthlies for the past year, and deletes the
remainder: