Rob Browning [Sat, 27 Jan 2018 21:19:24 +0000 (15:19 -0600)]
main: don't set stdio to nonblocking
This was left in the newliner replacement patch, and I can't remember
a justification for it. Having it in there affects other processes,
so take it out.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jan 2018 17:40:02 +0000 (11:40 -0600)]
Use absolute_import from the __future__ everywhere
Without this, among other things, we can end up with conflicts with
new upstream modules. For example, given lib/bup/io.py:
Traceback (most recent call last):
File "/home/rlb/src/bup/main-4/cmd/bup-index", line 10, in <module>
from bup import metadata, options, git, index, drecurse, hlinkdb
File "/home/rlb/src/bup/main-4/lib/bup/metadata.py", line 10, in <module>
from io import BytesIO
ImportError: cannot import name BytesIO
This switch also revealed a circular dependency between midx and git,
and several places where we weren't qualifying our bup imports
properly, i.e. "import git" rather than "from bup import git".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Jun 2016 20:49:14 +0000 (15:49 -0500)]
Clean subprocess output without newliner
Instead of launching a newliner subprocess from main.py to clean up the
output from other subprocesses (with respect to \r, shorter lines as
compared to the previous line, etc.), do something similar directly from
main.py via select, and then drop all of the attendant signal
forwarding, subprocess detaching via setsid(), etc. that the newliner
arrangement had accumulated.
Among other things, this should make bup much more responsive to
TERM/QUIT/etc. signals (i.e. C-c). Previously the
index/save/... subprocess might continue running even after bup
appeared to have quit.
To clean up the output, regularly check tty_width() and always write
"whole" space-padded lines, tracking stdout and stderr independently
(bup-newliner redirected both to stdout).
Rob Browning [Sun, 31 Dec 2017 18:23:01 +0000 (12:23 -0600)]
chain_ex: don't opportunistically add tracebacks to context_ex
Don't add a traceback to the context_ex while chaining. This could be
wrong, depending on the context, and in the common case, the traceback
more likely belonged on the result exception, not the context_ex.,
i.e.
except ... as ex:
raise chain_ex(ex, pending_ex)
Instead, just combine chain_ex with add_ex_tb when appropriate:
except ... as ex:
raise chain_ex(add_tb_ex(ex), pending_ex)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Without this "bup split -n NAME -d STAMP FILE" produces an error:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-split", line 67, in <module>
date = parse_date_or_fatal(opt.date, o.fatal)
NameError: name 'parse_date_or_fatal' is not defined
Signed-off-by: Fabian 'xx4h' Melters <xx4h@xx4h.de>
[rlb@defaultvalue.org: rebase on current master; adjust commit
message; fix commit author] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Nov 2017 19:24:41 +0000 (13:24 -0600)]
exc: support input arg; return SubprocInfo from exc and exo
Change exo to return a namedtuple so that you can refer to bits of the
result by name.
Base exc on exo so that it can support an input arg (even in python2),
and change both exc and exo to be a thinner wrapper around Popen so
that the arguments will behave roughly the same.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 13 Nov 2017 04:44:03 +0000 (22:44 -0600)]
vfs2: add trivial random eviction commit cache
Porting rm and gc to vfs2 has put us at the point where we need at
least some caching so that we're not invoking git an endless number of
times when trying to "bup rm" 1000 saves (i.e. via test-prune-older if
nothing else). Start with a very primitive, fixed-size commit cache
with random eviction.
It's not clear that the cache should always be enabled. While it may
help commands like rm, prune-older, fuse, web, ftp, etc., it may often
just be extra baggage for for commands like restore, ls, cat-file, ...
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 16:51:10 +0000 (10:51 -0600)]
vfs2: add tree_data_reader for use by cat-file
The cat-file command needs to be able to read arbitrary oids (chunked
or not) for --bupm, so add tree_data_reader for that purpose and
adjust fopen() to depend on it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 9 Dec 2017 00:21:21 +0000 (18:21 -0600)]
ls: add --commit-hash and drop vfs nominal_oid
Report the tree hash for commits for ls -s, unless --commit-hash is
also specified, and add some initial tests. This brings -s in line
with 38a49e9eb930e8a47e9de5a9715fad659a774e8b.
Rob Browning [Sat, 9 Dec 2017 19:46:13 +0000 (13:46 -0600)]
vfs2._resolve_path: improve handling ENOTDIR, absolute paths, etc.
Handle and test various path_resolution(7) cases more carefully.
e.g. ensure a trailing '/' or '/.' forces resolution, ensure a
non-final path element that's not a directory produces ENOTDIR, ...
Add tests for the these, for bad symlink handling, and a for number of
other cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:55:23 +0000 (15:55 -0500)]
vfs2: report tree metadata for commits
Don't synthesize metadata for a commit (like /foo, /foo/2017-..., or
/.tag/some-commit) by reporting the author date as the mtime, instead
use the metadata for the underlying commit tree.
Among other things, this avoids an inconsistency between the metadata
for ".../commit" and ".../commit/.".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Nov 2017 20:41:33 +0000 (15:41 -0500)]
vfs2: add fill_in_metadata_if_dir and ensure_item_has_metadata
Add fill_in_metadata_if_dir() as a contents() helper since contents()
does not return metadata for directories (because it's expensive).
This supports:
(name, fill_in_metadata_if_dir(repo, item)
for name, item in tuple(contents(...)))
Add ensure_item_has_metadata() to combine fill_in_metadata_if_dir()
and augment_item_meta() for cases like "ls -l" where you want
every item to have all the metadata available, and to have a Metadata
instance (even if faked), not just a mode.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 25 Nov 2017 17:10:03 +0000 (11:10 -0600)]
vfs2._resolve_path: require parent to be a path sequence
Require an actual path sequence for the parent since it provides the
context for the resolution of the path (with respect to relative
paths, symlinks containing .., etc.).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Oct 2017 19:40:15 +0000 (14:40 -0500)]
vfs2: suffix duplicate save names (e.g. 1970-01-01-214640-07)
From now on, append zero-padded integers to the names of saves that
have the same commit time in order to avoid duplicates. The sequence
number currently represents the save's reversed position in default
rev-list order, so that given:
Rob Browning [Sat, 11 Nov 2017 17:53:56 +0000 (11:53 -0600)]
test-rm: fix regexes to handle newer *and* older git
To accommodate optional lines, we need to use a construct like this:
foo(
bar)?
instead of this:
foo
(bar)?
The problem became apparent when people tested against older git after
the introduction of the patch to work with newer git: 292361d86d1cf0cc555681ae43371d66c8ebb366
Thanks to Greg Troxel and Basil Mohamed Gohar for reporting the
problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 21 Oct 2017 16:27:15 +0000 (11:27 -0500)]
test-rm: accommodate packed-refs creation by newer git
It looks like newer versions of git (as of at least 2.150 rc1) are
automatically creating packed-refs. Adjust test-rm.sh to accommodate
that possibility.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Sep 2017 17:08:44 +0000 (12:08 -0500)]
client: ensure server provides requested command
Gather the list of available commands from the server during client
initialization and use that to throw a suitable ClientError whenever
the server doesn't support a requested command.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
This will be more heavily used later and exceptions are notably more
expensive. Further, a missing object isn't necessarily an error.
It's perfectly reasonable to attempt to check an object's existence
via get().
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Sep 2017 20:08:33 +0000 (15:08 -0500)]
Use next(it), not it.next() and drop the helpers fallback
Change all it.next() invocations to next(it) for compatibility with
newer versions of python that do not support the next() method. Drop
the next() fallback helper since we depend on python 2.6 now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 11 Jul 2017 20:53:33 +0000 (15:53 -0500)]
test-meta: don't try to chattr +T a file
...since 'T' only works for directories, and newer kernels actually
reject the attempt (as of at least 4.12, and maybe 4.10). Thanks to
Mateusz Pavlic for reporting the problem and verifying the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 18 Jun 2017 20:40:41 +0000 (15:40 -0500)]
Support catpipe get(...,size=True); require git >= 1.5.6
Add a new size argument to get() that requests the object size in
addition to the type. To support this, require git 1.5.6 (circa 2008)
or newer so that we'll have cat-file --batch.
Remove the _slow_get() fallback since it's no longer needed.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 7 Apr 2017 00:32:50 +0000 (19:32 -0500)]
Support exception chaining and tracebacks
Python 3 has added support for exception chaining
https://www.python.org/dev/peps/pep-3134/
Which makes it possible to avoid losing information when an exception
is thrown from within an exception handler:
try:
...
raise lp0_on_fire()
except Exception as ex:
# Imagine the disk is also full and close() throws too
some_output_file.close()
In this situation, you'll never find out the printer's on fire. With
chaining, the first exception will be attached to the second as its
__context__. (Note that "finally" blocks suffer from the same issue.)
The PEP also describes adding a __traceback__ attribute to exceptions
so that they're more self-contained.
Python 3 handles all of this automatically, and includes any chained
exceptions in its tracebacks. For this:
$ python3 lib/bup/compat.py
Traceback (most recent call last):
File "lib/bup/compat.py", line 74, in outer
inner()
File "lib/bup/compat.py", line 70, in inner
raise Exception('first')
Exception: first
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "lib/bup/compat.py", line 78, in <module>
wrap_main(outer)
File "lib/bup/compat.py", line 50, in wrap_main
sys.exit(main())
File "lib/bup/compat.py", line 76, in outer
raise chain_ex(Exception('second'), ex)
Exception: second
Add a compat.py supporting something similar for Python 2, and use it
in main.py.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Ben Kelly [Tue, 14 Mar 2017 16:29:04 +0000 (12:29 -0400)]
prune-older: keep most recent save in each period group
Keep the most recent save in each period group (day, week, month, ...)
rather than the oldest.
Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log