bup.git
4 years agoAdd note/0.27.1-from-0.27.md and mention in README 0.27.1-rc1
Rob Browning [Sat, 30 Apr 2016 23:28:02 +0000 (18:28 -0500)]
Add note/0.27.1-from-0.27.md and mention in README

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 years agoMerge restore --sparse corruption fix
Rob Browning [Sat, 30 Apr 2016 23:24:15 +0000 (18:24 -0500)]
Merge restore --sparse corruption fix

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 years agorestore: fix --sparse corruption
Rob Browning [Sun, 24 Apr 2016 16:58:04 +0000 (11:58 -0500)]
restore: fix --sparse corruption

Rework the restore --sparse implementation to fix a bug that might cause
it to produce output that didn't match the original input.

Rewrite the code to probe for sparse regions in min_sparse_len jumps,
searching backward from the jump target for a non-zero byte.  If one is
found, move just past that byte and try again.

This should be substantially more efficient than the previous approach
for non-sparse regions.  In the limiting case, it should read roughly
1/min_sparse_len bytes instead of all of them.

The original bug was masked by the fact that the test that would have
revealed it wasn't actually generating random data (across trials),
something that was fixed in "Use $RANDOM seed for --sparse random
tests".

Thanks to Marcus Schopen for reporting the problem and Frank Gevaerts
for helping track down the solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 years agorestore: test --sparse with zeros at 64k boundary
Frank Gevaerts [Tue, 19 Apr 2016 20:02:49 +0000 (22:02 +0200)]
restore: test --sparse with zeros at 64k boundary

bup restore --sparse fails with short zero runs around 64K boundaries.
The new testcase reproduces that failure

Signed-off-by: Frank Gevaerts <frank@gevaerts.be>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit summary]

4 years agorestore: add generative --sparse testing
Rob Browning [Sun, 24 Apr 2016 22:52:02 +0000 (17:52 -0500)]
restore: add generative --sparse testing

Add t/sparse-test-data to automatically generate cases that might trip
up "restore --sparse", and use it in test-sparse-files.sh.  This change
causes the current --sparse code to fail if you run enough trials,
but not as quickly as the existing "bup random" based test.

As compared to the existing "bup random" test, this one should evaluate
larger sparse regions much more often.

Thanks to Frank Gevaerts for help evaluating the fix.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 years agoUse $RANDOM seed for --sparse random tests
Rob Browning [Sun, 24 Apr 2016 22:07:39 +0000 (17:07 -0500)]
Use $RANDOM seed for --sparse random tests

Since bup random defaults to a fixed random seed of 1, the restore
--sparse test for random data was actually testing the same data every
time.  Specify a $RANDOM --seed to fix that.

After this change, the affected test fails fairly often.

Thanks to Frank Gevaerts for help evaluating the fix.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoHACKING: update for 0.27 release 0.27
Rob Browning [Sun, 26 Apr 2015 16:38:57 +0000 (11:38 -0500)]
HACKING: update for 0.27 release

5 years agoMakefile: no really, run test.sh (fix subst too) 0.27-rc7
Rob Browning [Sat, 25 Apr 2015 16:02:38 +0000 (11:02 -0500)]
Makefile: no really, run test.sh (fix subst too)

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoREADME.md: mention workaround for pwd test issue 0.27-rc6
Rob Browning [Sat, 25 Apr 2015 15:30:30 +0000 (10:30 -0500)]
README.md: mention workaround for pwd test issue

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: fix test pattern to include t/test.sh
Rob Browning [Sat, 25 Apr 2015 15:22:33 +0000 (10:22 -0500)]
Makefile: fix test pattern to include t/test.sh

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-xdev: show test skip messages via WVSTART
Rob Browning [Sat, 25 Apr 2015 15:13:14 +0000 (10:13 -0500)]
test-xdev: show test skip messages via WVSTART

Otherwise they're invisible via "make check", etc.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoReject invalid string in --date argument
Johannes Berg [Thu, 23 Apr 2015 20:41:45 +0000 (22:41 +0200)]
Reject invalid string in --date argument

As parse_date_or_fatal() currently uses atof(), which just returns 0
if the string isn't a valid number, it can never actually be fatal
and will just use "1970-01-01 00:00:00" as the time if the string is
specified wrong.

Fix that by using float() directly so ValueError() is raised.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoIgnore lchmod() ENOSYS (i.e. function not implemented) 0.27-rc5
Rob Browning [Thu, 11 Sep 2014 18:52:04 +0000 (13:52 -0500)]
Ignore lchmod() ENOSYS (i.e. function not implemented)

When restoring symlink permissions, assume that ENOSYS means that it's
not possible to lchmod().

For comparison, it looks like tar doesn't handle lchmod() at all yet.
From the Debian tar 1.27.1 TODO:

  * Add support for restoring the attributes of symbolic links, for
    OSes like FreeBSD that have the lutimes and lchmod functions.

Thanks to Xiao LIU for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoconfigure-sampledata: remove vestigial etc too
Rob Browning [Sat, 11 Apr 2015 17:10:38 +0000 (12:10 -0500)]
configure-sampledata: remove vestigial etc too

Thanks to Mark J Hewitt for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoDon't use "cp -a"
Rob Browning [Sun, 29 Mar 2015 16:47:45 +0000 (11:47 -0500)]
Don't use "cp -a"

It's not portable -- use portable cp options, or a new t/sync-tree
instead.  Thanks to Greg Troxel for reporting the problem after testing
on NetBSD 5.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoGet TZ offset from C localtime, given tm_gmtoff 0.27-rc4
Rob Browning [Sat, 28 Mar 2015 20:09:47 +0000 (15:09 -0500)]
Get TZ offset from C localtime, given tm_gmtoff

If we detect that struct tm contains tm_gmtoff, use the system
localtime() to compute timezone offsets.  This may help fix problems on
platforms where Python strftime "%z" doesn't report accurate timzeone
information.

Thanks to Patrick Rouleau for reporting just such a problem on Cygwin.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMerge --sparse restore fix and test improvements
Rob Browning [Sat, 28 Mar 2015 17:56:28 +0000 (12:56 -0500)]
Merge --sparse restore fix and test improvements

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoAdjust sparse restore tests for test fs block size
Rob Browning [Sat, 28 Mar 2015 17:48:19 +0000 (12:48 -0500)]
Adjust sparse restore tests for test fs block size

Change test-sparse-files.sh to detect the test fs block size (when
possible) and adjust its behavior accordingly.  If the block size can't
be determined, use a block size of 3MB, which is hoped to be larger than
any block sizes we'll encounter anytime soon.

Previously the tests might fail on filesystems with relatively large
block sizes, like those on the current Debian powerpc and ppc64el build
daemons (64k).

Thanks to Goswin Brederlow for mentioning that the Lucene block size is
1MB, to Robert Edmonds for running a build through the Debian buildds,
which revealed the problem, and to Julien Cristau for reporting the
block size on the failing buildds.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoRework write_sparsely() to fix in-buffer zero runs
Rob Browning [Sun, 15 Mar 2015 21:43:47 +0000 (16:43 -0500)]
Rework write_sparsely() to fix in-buffer zero runs

Fix the sparse restoration of buffers that have non-zero bytes, followed
by a run of zero bytes that's longer than the minimum sparse run
length (currently 512), followed by non-zero bytes.

Previously, the initial non-zero bytes would be *lost*.

In the new code, don't unconditionally output previous zero bytes --
merge them with any leading zeros in the current block.

And allow arbitrarily large sparse regions; use append_sparse_region()
to break up runs that are too large for off_t into a sequence of seeks
of no more than INT_MAX bytes each.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoTest sparse restore of short in-buffer zero run
Rob Browning [Sun, 15 Mar 2015 21:38:34 +0000 (16:38 -0500)]
Test sparse restore of short in-buffer zero run

Test that sparse --restore handles the case where within one call to
write_sparsely() (one buffer) we have non-zero bytes, followed by a run
of zero bytes that's longer than the minimum sparse run
length (currently 512), followed by non-zero bytes.

Currently, the initial non-zero bytes will be lost, and this test will
fail.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoUse t/sampledata, not make install, for tests
Rob Browning [Sat, 21 Mar 2015 22:17:12 +0000 (17:17 -0500)]
Use t/sampledata, not make install, for tests

This should be more efficient, and is intended fix the problems people
have experienced with the recursive "make install" invocations that some
tests were using to produce input data.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoOnly build _version.py once; remove phony targets
Rob Browning [Sat, 21 Mar 2015 19:32:30 +0000 (14:32 -0500)]
Only build _version.py once; remove phony targets

Previously _version.py was a phony target because we couldn't easily
tell when the current git working tree version had changed.  This caused
various targets to be rebuilt multiple times (i.e. recursive make
invocations, etc.).

To fix that, just update _version.py once (at startup) if needed, via an
immediate variable assignment that calls a new ./configure-version
command, i.e.

  initial_setup := $(shell ./configure-version --update)

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoCreate t/sampledata/var/ and version it
Rob Browning [Sat, 21 Mar 2015 19:57:18 +0000 (14:57 -0500)]
Create t/sampledata/var/ and version it

Maintain a new t/sampledata/var/ (via t/configure-sampledata) that
contains any test data that we don't want to or can't commit to
git (i.e. symlinks, and other dynamically generated data).  Move all of
the existing generated data there, and delete var/ entirely on clean.

Control the creation of var/ with make, via the existence of
t/sampledata/var/rev/vN.  Whenever we change the content, we'll change
N (currently 0), which will force the directory to be recreated.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoDon't have ./bup depend on main.py, etc.
Rob Browning [Sat, 21 Mar 2015 19:40:48 +0000 (14:40 -0500)]
Don't have ./bup depend on main.py, etc.

Instead create bup_deps (which includes bup) and have everything that
needs a functional ./bup depend on that set of targets.

Previously bup would often be rebuilt unnecessarily (even if it didn't
depend on other phony targets) because it depended on _version.py and
main.py.  Since bup ends up symlinked to the latter, and make looks at
symlink target timestamps, and _version.py would often be newer than
main.py, ./bup would be repeatedly rebuilt.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoClean up sampledata/abs-symlink
Rob Browning [Sat, 21 Mar 2015 20:19:52 +0000 (15:19 -0500)]
Clean up sampledata/abs-symlink

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosubtree-hash: adjust for python 2.6
Rob Browning [Thu, 19 Mar 2015 04:56:24 +0000 (23:56 -0500)]
subtree-hash: adjust for python 2.6

Don't use argparse or check_output, use options and helpers.readpipe
instead.

Thanks to enverex for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoUse *native* type sizes in file attr guard
Rob Browning [Sun, 15 Mar 2015 15:34:38 +0000 (10:34 -0500)]
Use *native* type sizes in file attr guard

I'd misunderstood what Python struct meant by the *standard* size, and
what I'd wanted in the guard was the native size, so ask for that (via
'@').

Thanks to Robert Edmunds for running a build through the Debian buildds
and then reporting the problem that revealed on s390x.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: fix -j race to create ./bup 0.27-rc3
Rob Browning [Sat, 14 Mar 2015 15:39:19 +0000 (10:39 -0500)]
Makefile: fix -j race to create ./bup

Previously the bup target might fail if it was run in parallel
(i.e. during make -j).  Since bup depends on _version.py, which is
phony, that wasn't unlikely.

Fix the race by ignoring errors while creating the symlink, and then
testing for existence afterward.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosubtree-hash: handle non-ASCII path chars 0.27-rc2
Rob Browning [Fri, 13 Mar 2015 02:41:50 +0000 (21:41 -0500)]
subtree-hash: handle non-ASCII path chars

Pass "-z" to git ls-files to turn off git's core.quotepath, so we can
handle non-ASCII path characters directly, and rewrite the code in
python to make that easier.

Thanks to Jiří Martínek for reporting the issue -- that
test-redundant-saves.sh would fail when there was a "ý" character in the
source tree path.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMerge pull request #13 from Farioko/patch-1
Gabriel Filion [Tue, 10 Mar 2015 00:24:24 +0000 (20:24 -0400)]
Merge pull request #13 from Farioko/patch-1

Correct grammar in bup save -r example.

5 years agoMakefile: clean up _version.py temp files
Rob Browning [Mon, 9 Mar 2015 03:34:44 +0000 (22:34 -0500)]
Makefile: clean up _version.py temp files

Thanks to Gabriel Filion for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMerge the "bup import-duplicity" command
Rob Browning [Mon, 9 Mar 2015 02:16:24 +0000 (21:16 -0500)]
Merge the "bup import-duplicity" command

Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoimport-duplicity: document as EXPERIMENTAL
Rob Browning [Mon, 9 Mar 2015 02:11:28 +0000 (21:11 -0500)]
import-duplicity: document as EXPERIMENTAL

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoPort import-duplicity to python
Rob Browning [Sun, 22 Feb 2015 22:42:18 +0000 (16:42 -0600)]
Port import-duplicity to python

Additionally, specify an --archive-dir to all duplicity invocations so
that we don't scribble in ~/.cache/duplicity during the imports, and
unconditionally clean up temporary dirs.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-import-duplicity: use install tree for data
Rob Browning [Mon, 9 Mar 2015 01:59:06 +0000 (20:59 -0500)]
test-import-duplicity: use install tree for data

Use a stable "make install" tree for the backup data, and test the
restoration via compare-trees instead of diff.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-import-duplicity: clean up; add --archive-dir
Rob Browning [Sun, 22 Feb 2015 21:27:43 +0000 (15:27 -0600)]
test-import-duplicity: clean up; add --archive-dir

If we don't specify an --archive-dir to duplicity, it'll scribble in
~/.cache/duplicity during testing.  This is only a partial fix, given
the duplicity invocations inside import-duplicity itself.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoimport-duplicity: fix typos
Rob Browning [Sun, 22 Feb 2015 21:33:46 +0000 (15:33 -0600)]
import-duplicity: fix typos

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoQuote command arguments in import-duplicity-cmd.sh.
Rob Browning [Sun, 7 Oct 2012 18:57:09 +0000 (13:57 -0500)]
Quote command arguments in import-duplicity-cmd.sh.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoCatch ^C in import-duplicity
Zoran Zaric [Mon, 24 Sep 2012 23:13:33 +0000 (01:13 +0200)]
Catch ^C in import-duplicity

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
5 years agoMake import-duplicity's output more verbose
Zoran Zaric [Mon, 24 Sep 2012 23:13:32 +0000 (01:13 +0200)]
Make import-duplicity's output more verbose

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
5 years agoAdd documentation for import-duplicity
Zoran Zaric [Mon, 24 Sep 2012 23:13:31 +0000 (01:13 +0200)]
Add documentation for import-duplicity

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
5 years agoAdd a testcase for the import-duplicity command
Zoran Zaric [Mon, 24 Sep 2012 23:13:30 +0000 (01:13 +0200)]
Add a testcase for the import-duplicity command

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: adjust commit message; update tests during rebase
 and move them from test.sh to t/test-import-duplicity.sh following
 break-up of test.sh.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoImplement an import-duplicity command
Zoran Zaric [Mon, 24 Sep 2012 23:13:29 +0000 (01:13 +0200)]
Implement an import-duplicity command

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: adjust commit message]

5 years agocompare-trees: redirect incidental info to stderr
Rob Browning [Mon, 9 Mar 2015 01:49:49 +0000 (20:49 -0500)]
compare-trees: redirect incidental info to stderr

Leave stdout for the rsync --itemize-changes output, so we can use it in
tests.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: support parallel testing
Rob Browning [Sun, 8 Mar 2015 23:07:23 +0000 (18:07 -0500)]
Makefile: support parallel testing

These changes allow "make -j check" to work, which substantially
decreases the testing time here.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: remove default target in favor of all
Rob Browning [Sat, 21 Feb 2015 22:15:12 +0000 (16:15 -0600)]
Makefile: remove default target in favor of all

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: make t/tmp a real target
Rob Browning [Sat, 21 Feb 2015 22:03:14 +0000 (16:03 -0600)]
Makefile: make t/tmp a real target

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: remove vestigial bup-% and % targets
Rob Browning [Sat, 21 Feb 2015 21:57:35 +0000 (15:57 -0600)]
Makefile: remove vestigial bup-% and % targets

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agoMakefile: eliminate phony/duplicate bup target
Rob Browning [Sat, 21 Feb 2015 21:53:42 +0000 (15:53 -0600)]
Makefile: eliminate phony/duplicate bup target

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-fsck.sh: use stable data for tests
Rob Browning [Sun, 8 Mar 2015 21:02:28 +0000 (16:02 -0500)]
test-fsck.sh: use stable data for tests

Use a stable install tree plus t/sampledata (for some variety) for the
test data, instead of "whatever's in ./lib, ./Documentation, and ./cmd".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotindex.py: move test file foo to t/tmp
Rob Browning [Sun, 8 Mar 2015 20:38:32 +0000 (15:38 -0500)]
tindex.py: move test file foo to t/tmp

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosampledata: use a local absolute symlink, not /etc
Rob Browning [Sun, 8 Mar 2015 20:37:20 +0000 (15:37 -0500)]
sampledata: use a local absolute symlink, not /etc

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoAlways pass --always to git-describe
Rob Browning [Sun, 1 Feb 2015 21:08:02 +0000 (15:08 -0600)]
Always pass --always to git-describe

Without this the build might fail if the git history is short enough,
perhaps via a shallow clone.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agot/test-on.sh: Don't quote "wc -l" output
Alexander Barton [Wed, 25 Feb 2015 14:28:21 +0000 (15:28 +0100)]
t/test-on.sh: Don't quote "wc -l" output

Some wc(1) implementations, for example on OS X, prefix the numeric
output with spaces. So don't quote this output (and don't treat it
as a string) to not confuse WVPASSEQ.

This fixes the following error of "make check":

Comparing:
       2
--
2
! t/test-on.sh:25  ' 2' = '2'                     FAILED
 2.108s ok
called from t/test-on.sh:25 WVPASSEQ        2 2
make[1]: *** [runtests-cmdline] Error 1
! Program returned non-zero exit code (2)         FAILED
WvTest: 3007 tests, 1 failure, total time 84.013s.

Reported by Pedro Estarque, thanks!

Signed-off-by: Alexander Barton <alex@barton.de>
[rlb@defaultvalue.org: adjust commit message; add WVPASS calls]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agovfs.py: remove redundant assignment
Holger Dell [Tue, 24 Feb 2015 15:08:29 +0000 (16:08 +0100)]
vfs.py: remove redundant assignment

Signed-off-by: Holger Dell <github@holgerdell.com>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoweb: URL escape link components 0.27-rc1
Rob Browning [Mon, 16 Feb 2015 19:13:09 +0000 (13:13 -0600)]
web: URL escape link components

So that links for path names like directory:80 won't be broken.

Thanks to Euan Kemp for reporting the problem and proposing an earlier
fix.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoweb: don't double-escape title path
Rob Browning [Sun, 15 Feb 2015 20:15:57 +0000 (14:15 -0600)]
web: don't double-escape title path

Tornado escapes {{}} content via xhtml_escape by default, so drop the
explicit (extra) escape() call.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosave: make --strip-path=/ a no-op
Rob Browning [Sat, 7 Feb 2015 16:49:01 +0000 (10:49 -0600)]
save: make --strip-path=/ a no-op

When --strip-path=/ is specified, don't treat it as
--strip-path=/{usr,lib,...}, instead do nothing.

Thanks to Mark J Hewitt for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agohlinkdb.py: clean up temp file more carefully
Rob Browning [Thu, 17 Jul 2014 17:47:43 +0000 (12:47 -0500)]
hlinkdb.py: clean up temp file more carefully

Make sure to always close the temp file, and if something goes wrong
while preparing the save, make sure to delete it and remove it from
consideration by commit.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: don't test for bup.1
Rob Browning [Sat, 7 Feb 2015 15:27:22 +0000 (09:27 -0600)]
test-meta.sh: don't test for bup.1

The manpages may not exist (i.e. if pandoc's not available).

Thanks to Mark J Hewitt for reporting the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoon: handle remote stdout and stderr via mux
Rob Browning [Sun, 9 Nov 2014 00:32:55 +0000 (18:32 -0600)]
on: handle remote stdout and stderr via mux

Previously, when running "bup on" (i.e. when in BUP_SERVER_REVERSE
mode), bup would redirect the remote command's stdout to stderr because
stderr is the only remaining avenue back to the local console.  It's the
only avenue because in reverse mode, stdout (and stdin) are connected
back to a local "bup server" instance (hence the "reverse").

Of course that makes it impossible to reliably capture the
non-diagnostic output from the remote commands, i.e.

  commit_id="$(bup on HOST save -t ...)"

To fix that, in the remote "bup on--server" multiplex the stdout and
stderr from all "bup on" subcommands with "bup mux", and then
demultiplex those streams back to the local stdout and stderr via
DemuxConn() in the receiving "bup on".

Thanks to Alexander Barton for pointing out an error in a previous
version of this commit message, and thanks to Gabriel Filion for
pointing out that we could use the existing mux infrastructure instead
of reinventing the wheel.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agomux: give stdin exclusively to subcommand
Rob Browning [Tue, 11 Nov 2014 22:23:22 +0000 (16:23 -0600)]
mux: give stdin exclusively to subcommand

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: use a tmpdir, not ./bupmeta.tmp
Rob Browning [Mon, 19 Jan 2015 20:50:26 +0000 (14:50 -0600)]
test-meta.sh: use a tmpdir, not ./bupmeta.tmp

Use a real wvmktempdir created directory rather than the top-level
./bupmeta.tmp.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: move mount points to t/mnt
Rob Browning [Mon, 19 Jan 2015 19:36:32 +0000 (13:36 -0600)]
test-meta.sh: move mount points to t/mnt

When root, don't create test mount points in ./bupmeta.tmp, use
t/wvmkmountpt.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosave: make save tree collisions a deferred error
Rob Browning [Thu, 30 Oct 2014 15:40:28 +0000 (10:40 -0500)]
save: make save tree collisions a deferred error

Check for duplicate names when saving a tree, and when duplicates are
found, issue a deferred error, keep the first, and ignore the rest.

Duplicates/collisions can be introduced by the strip/graft options.
For example, given:

  /x/1
  /y/1

consider this command:

  bup save --strip /x /y

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosave: always push parents when entering a subtree
Rob Browning [Tue, 4 Nov 2014 18:09:37 +0000 (12:09 -0600)]
save: always push parents when entering a subtree

Once we enter the subtree, all of the parents must be on the stack for
subsequent operations to work correctly (like the _pop() in the
following "if not file:" section).

Consider the case where /some/empty/dir was indexed, but removed before
save.  Without this fix, the metadata read for "dir" would fail and the
pending directory stack (parts) would include /some/empty instead of
/some/empty/dir.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agosave-cmd.py: remove redundant _push()
Rob Browning [Mon, 3 Nov 2014 23:02:03 +0000 (17:02 -0600)]
save-cmd.py: remove redundant _push()

Remove a duplicate _push() from the "first_root" code.  This could cause
the creation of a tree that was immediately _pop()ped by the "finish the
current sub-tree" code, and then reintroduced (as a duplicate parent
entry) by the "start a new sub-tree" code.

Instead, just wait for the "start a new sub-tree" code to call the
relevant _push().

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotmetadata.py: print saved_errors in from_path test
Rob Browning [Sat, 31 Jan 2015 18:06:26 +0000 (12:06 -0600)]
tmetadata.py: print saved_errors in from_path test

And prefix saved_errors with 'saved_errors:' in both
test_from_path_error and test_apply_to_path_restricted_access.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-fuse: unset BLOCK_SIZE, etc. for ls output
Rob Browning [Sat, 31 Jan 2015 16:41:35 +0000 (10:41 -0600)]
test-fuse: unset BLOCK_SIZE, etc. for ls output

If any of the environment variables that this patch removes are set, ls
may not provide the expected test output.

Thanks to Koen Roossen for reporting the problem, and suggesting the
solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoDon't fadvise we don't need data that we do need
Aidan Hobson Sayers [Sun, 25 Jan 2015 23:50:27 +0000 (23:50 +0000)]
Don't fadvise we don't need data that we do need

The documentation for posix_fadvise states that if len is 0 the advice
extends to the end of the file. We currently always pass 0 for
the first two posix_fadvise calls (first because ofs is 0, second
because ofs == BLOB_READ_SIZE == 1024*1024) so we're advising the kernel
to dump any predicted file caching twice per file.

This patch ensures we don't pass a len of 0 in the two scenarios above.

Signed-off-by: Aidan Hobson Sayers <aidanhs@cantab.net>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: handle BUP_DIR via wvmktempdir
Rob Browning [Sun, 18 Jan 2015 21:58:02 +0000 (15:58 -0600)]
test-meta.sh: handle BUP_DIR via wvmktempdir

Put the BUP_DIR in a real tmpdir instead of ./buptest.tmp.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotgit.py: set git committer/author info
Rob Browning [Mon, 19 Jan 2015 18:57:44 +0000 (12:57 -0600)]
tgit.py: set git committer/author info

Without this, the tests will fail if you don't have a user.name and
user.email in your git configuration (i.e. in a test VM).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoon: always catch our exceptions
Rob Browning [Thu, 13 Nov 2014 19:11:49 +0000 (13:11 -0600)]
on: always catch our exceptions

Thanks to Gabriel Filion for reporting the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoon: only wait for a process if it might exist
Rob Browning [Thu, 13 Nov 2014 18:17:23 +0000 (12:17 -0600)]
on: only wait for a process if it might exist

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoadd Arch Linux to list of known OSs with binary packages
Thiago Barroso Perrotta [Tue, 6 Jan 2015 21:49:15 +0000 (19:49 -0200)]
add Arch Linux to list of known OSs with binary packages

5 years agoSkip dependent tests if we can't load loop or fuse
Rob Browning [Thu, 20 Nov 2014 16:47:58 +0000 (10:47 -0600)]
Skip dependent tests if we can't load loop or fuse

When modprobe is available (or when we know we're on Linux), try to load
the loop(back) and fuse modules before using them, and skip the relevant
tests if the load fails.  This allows the tests to proceed on systems
lacking those modules.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoCall add_error() with one arg on readlink failure
Gabriel Filion [Thu, 20 Nov 2014 22:07:11 +0000 (17:07 -0500)]
Call add_error() with one arg on readlink failure

...instead of two.

This bug and its solution was found by Ben Kelly.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
5 years agoindex: Correct inconsitency in documentation of modes
Gabriel Filion [Wed, 19 Nov 2014 17:30:07 +0000 (12:30 -0500)]
index: Correct inconsitency in documentation of modes

In the optspec, --clear and --check are documented as being modes, but
in the man page they are simple options.

Also, even though --clear and --check are listed as modes in the
optspec, they don't appear in the synopsis line.

According to the code, --clear and --check do behave as modes so we
should be consistent in how we document them.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoCorrect claim about number of packs per backup
Gabriel Filion [Mon, 17 Nov 2014 17:24:50 +0000 (12:24 -0500)]
Correct claim about number of packs per backup

The current sentence implies that there is only one pack file per backup
run, which is not necessarily correct.

Because of the "constants" max_pack_size and max_pack_objects that are
used as limits to pack sizes, we may end up with more than one pack per
backup if there is a lot of data that needs to be stored.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
5 years agotag: remove docstring about unexistant feature
Gabriel Filion [Mon, 17 Nov 2014 17:24:49 +0000 (12:24 -0500)]
tag: remove docstring about unexistant feature

This was the intention for first creating the subcommand but since we
still don't have a means of removing old data, this docstring is talking
about a ghost.

I'll remember to push for that use case when we implement data removal.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
5 years agotag-cmd.py: fix -f placement in delete synopsis
Gabriel Filion [Sat, 15 Nov 2014 22:06:11 +0000 (17:06 -0500)]
tag-cmd.py: fix -f placement in delete synopsis

The -d option is the one that should take the <tag name> argument,
whereas the -f option has no argument.

With better ordering in the synopis, this is less misleading.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
5 years agotag-cmd.py: rephrase -f summary in optspec
Gabriel Filion [Sat, 15 Nov 2014 22:06:10 +0000 (17:06 -0500)]
tag-cmd.py: rephrase -f summary in optspec

The current summary for the -f option is rather unclear: it says that it
will delete a tag that doesn't exist, but will happily delete the tag,
too, if it exists.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
5 years agotag-cmd.py: use git update-ref to delete tags
Gabriel Filion [Sat, 15 Nov 2014 22:06:08 +0000 (17:06 -0500)]
tag-cmd.py: use git update-ref to delete tags

Sometimes tags can be elsewhere than in the refs/tags/ directory. This
can happen if git pack-refs --all was used.

The current code chokes on such a situation. Since git update-ref
already knows about this subtlety, let it handle the deletion.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agomux: support wrapping arbitrary commands
Rob Browning [Tue, 11 Nov 2014 18:09:04 +0000 (12:09 -0600)]
mux: support wrapping arbitrary commands

Don't just support bup commands; allow arbitrary commands as arguments,
i.e. instead of

  bup mux BUP_SUBCOMMAND ...

support

  bup mux COMMAND ...

i.e.

  bup mux tar cvf ...

Reviewed-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: rebased to master; removed not yet applicable
 on--server-cmd.py changes]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-on.sh: exit only once on source failure
Rob Browning [Wed, 12 Nov 2014 18:34:50 +0000 (12:34 -0600)]
test-on.sh: exit only once on source failure

Thanks to Gabriel Filion for pointing out the oversight.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoTest for load failures in t/test-*.sh
Rob Browning [Mon, 10 Nov 2014 03:50:02 +0000 (21:50 -0600)]
Test for load failures in t/test-*.sh

i.e.

  . foo || exit $?

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agogit.py: generalize update_ref() to support tags in addition to heads
Rob Browning [Sun, 23 Mar 2014 18:10:01 +0000 (13:10 -0500)]
git.py: generalize update_ref() to support tags in addition to heads

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agogit.py: allow the specification of a repo_dir to update_ref()
Rob Browning [Sun, 23 Mar 2014 18:07:25 +0000 (13:07 -0500)]
git.py: allow the specification of a repo_dir to update_ref()

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoAllow the specification of a repo_dir to some VFS and git operations
Rob Browning [Wed, 22 Jan 2014 18:55:16 +0000 (12:55 -0600)]
Allow the specification of a repo_dir to some VFS and git operations

Previously, these VFS and git operations would only operate on the
default repository (git.repo()).

Have vfs.cp() handle more than one repository (via an internal cache).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoAllow per-instance specification of the repo_dir for CatPipe
Rob Browning [Tue, 21 Jan 2014 23:31:13 +0000 (17:31 -0600)]
Allow per-instance specification of the repo_dir for CatPipe

This allows the creation of CatPipes that refer to repositories other
than the default, which in turn allows copying between repositories,
etc.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: add missing "$() || exit $?" tests
Rob Browning [Tue, 14 Oct 2014 20:40:06 +0000 (15:40 -0500)]
test-meta.sh: add missing "$() || exit $?" tests

Their omission was masking bugs fixed in several of the previous
commits.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoChange incorrect unknown-owners references
Rob Browning [Tue, 14 Oct 2014 20:28:59 +0000 (15:28 -0500)]
Change incorrect unknown-owners references

The test program is actually called unknown-owner (t/unknown-owner).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: escape user/group expressions
Rob Browning [Tue, 14 Oct 2014 17:20:14 +0000 (12:20 -0500)]
test-meta.sh: escape user/group expressions

Escape external user/group strings before passing them to 'grep -E',
otherwise calls like this will fail when the string includes regular
expression metacharacters:

Thanks to Pierre-Emmanuel Novac <piernov@gmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-meta.sh: finish renaming variables
Rob Browning [Tue, 14 Oct 2014 17:22:51 +0000 (12:22 -0500)]
test-meta.sh: finish renaming variables

Apparently the changes started here weren't completely finished;
finish them:

  b836113e503ab3f8e3c30c3283eb59f936fb0023

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agotest-cat-file.sh: handle hashsplit bupm files
Rob Browning [Fri, 17 Oct 2014 15:44:24 +0000 (10:44 -0500)]
test-cat-file.sh: handle hashsplit bupm files

A bit back we started allowing .bupm files to be hashsplit:

  ee777400cf624b4f7a24b15614c2b6c745560dcb

So add t/git-cat-tree and use it in test-cat-file.sh to prevent the
test from crashing whenever the bupm is a tree.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
5 years agoAdd support for "restore --sparse"
Rob Browning [Thu, 16 Jan 2014 00:39:09 +0000 (18:39 -0600)]
Add support for "restore --sparse"

Thanks to Patrick Rouleau <prouleau72@gmail.com> for spotting a
serious potential short-write problem in an earlier revision of this
work, and suggesting other improvements.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
6 years agosave: catch and log SSH errors
Gabriel Filion [Thu, 4 Sep 2014 06:53:14 +0000 (02:53 -0400)]
save: catch and log SSH errors

Currently, if the ssh connection to the remote server fails, we get the
server errors, then an ugly traceback that ends up saying "look at the
errors above". It makes it look like the traceback is part of the errors
that we need to look at, when in reality it's just an exception that's
not handled correctly by bup-save.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
6 years agoFix typo in atomically_replaced_file docstring
Rob Browning [Fri, 29 Aug 2014 20:32:21 +0000 (15:32 -0500)]
Fix typo in atomically_replaced_file docstring

Thanks to Gabriel Filion <gabster@lelutin.ca> for pointing out the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
6 years agoAssume FS_IOC_GETFLAGS may trash output on error
Mark J Hewitt [Mon, 11 Aug 2014 10:36:19 +0000 (11:36 +0100)]
Assume FS_IOC_GETFLAGS may trash output on error

Don't assert that the attr value is reasonable (<= UINT_MAX) when
FS_IOC_GETFLAGS returns an error, because it may change the the value
to something invalid in that situation.

This has been observed to occur when, for example, the underlying
filesystem does not support the operation,

Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
6 years agoHandle kernel/FUSE disagreement over Linux attrs
Rob Browning [Tue, 29 Jul 2014 20:35:07 +0000 (15:35 -0500)]
Handle kernel/FUSE disagreement over Linux attrs

Use long for the Linux attr type, so that it should work on all
little-endian systems, across both normal and FUSE-backed filesystems.

Disable Linux attr support for now on big-endian systems where
sizeof(long) > sizeof(int).  See the changes to bup-index.md and
_helpers.c for more information.

Eventually, we may want to add an argument to allow re-enabling
support on affected systems, when the user can guarantee a homogeneous
filesystem type -- or better yet, the kernel and FUSE will finally
sort out their issues.

Thanks to Tilo Schwarz <mail@tilo-schwarz.de> and daryl5@arcor.de for
reporting and pursuing the problems that lead to this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>