Rob Browning [Sat, 11 Jun 2016 17:59:38 +0000 (12:59 -0500)]
test-web: use path with unique unicode sequence
Change the test path from "éxcitement" to "¡excitement!" since the
former has multiple valid unicode encodings (e.g. given combining
characters, etc.).
With the previous string, the test might fail if something along the way
normalized it (which was the case on at least some versions of OS X).
Thanks to ilovezfs for reporting the problem and helping with the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 27 May 2016 00:30:40 +0000 (19:30 -0500)]
Fix release archives and add tests
Previously, an archive created by "git archive" would fail to build,
because the versioning code was broken (in part by 03d35993b87f49753542e705e37949a46964be17) for any build outside a git
clone. Although "bup version" may not really have been quite right
since the switch from "bup-VERSION" to "VERSION release tags.
Fix the problem by reintroducing the content of the older
_version.py.pre as a permanent _release.py file, which (as with the
previous incarnation) is a template that's expanded during "git archive"
via export-subst (cf. gitattributes(5)).
Add a version module as the public interface to versioning, and have it
sort out whether or not the current tree is an official release. Adjust
configure-version to manage _checkout.py instead of _version.py (drop
_version.py), so that the version module is effectively just
disambiguating between _release.py and _checkout.py.
Adjust "bup version" to use the new version module and to handle
unprefixed VERSION tags.
Finally, add test-release-archive.sh, invoked via "make distcheck", so
we (hopefully) don't unknowingly break this again.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 28 May 2016 14:37:09 +0000 (09:37 -0500)]
Add config.vars.sh for bup_make and bup_python
During configure create a config/config.vars.sh file including (for now)
properly shell escaped bup_make and bup_python definitions so that we
can use them in other schell scripts.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 May 2016 00:15:30 +0000 (19:15 -0500)]
Compute exact sizes via t/data-size in test-gc.sh
Add a t/data-size helper that computes the total data size of all named
paths and use it instead of du in test-gc.sh to determine the
before/after repository sizes.
This should fix spurious test failures on filesystems with a large block
size. (Observed on ppc64el and powerpc tmpfs filesystems.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 16:58:04 +0000 (11:58 -0500)]
restore: fix --sparse corruption
Rework the restore --sparse implementation to fix a bug that might cause
it to produce output that didn't match the original input.
Rewrite the code to probe for sparse regions in min_sparse_len jumps,
searching backward from the jump target for a non-zero byte. If one is
found, move just past that byte and try again.
This should be substantially more efficient than the previous approach
for non-sparse regions. In the limiting case, it should read roughly
1/min_sparse_len bytes instead of all of them.
The original bug was masked by the fact that the test that would have
revealed it wasn't actually generating random data (across trials),
something that was fixed in "Use $RANDOM seed for --sparse random
tests".
Thanks to Marcus Schopen for reporting the problem and Frank Gevaerts
for helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 22:52:02 +0000 (17:52 -0500)]
restore: add generative --sparse testing
Add t/sparse-test-data to automatically generate cases that might trip
up "restore --sparse", and use it in test-sparse-files.sh. This change
causes the current --sparse code to fail if you run enough trials,
but not as quickly as the existing "bup random" based test.
As compared to the existing "bup random" test, this one should evaluate
larger sparse regions much more often.
Thanks to Frank Gevaerts for help evaluating the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 22:07:39 +0000 (17:07 -0500)]
Use $RANDOM seed for --sparse random tests
Since bup random defaults to a fixed random seed of 1, the restore
--sparse test for random data was actually testing the same data every
time. Specify a $RANDOM --seed to fix that.
After this change, the affected test fails fairly often.
Thanks to Frank Gevaerts for help evaluating the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
stevelr [Sun, 7 Feb 2016 02:05:51 +0000 (18:05 -0800)]
Add add_error imports missed in import * removal
[rlb@defaultvalue.org: rebased on current master; squashed related
patches; adjusted commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 27 Mar 2016 00:33:57 +0000 (19:33 -0500)]
Check saved_errors before and after every test
Add a new buptest module that includes a no_lingering_errors context
manager which provokes a wvtest failure if there are any saved_errors
before or after a test, and then clears them, and use that manager in
all of the Python tests so that unexpected errors will be detected
immediately, instead of causing an unrelated failure later, when testing
for expected saved_errors.
Add a temp_testdir context manager to replace all of the current
tmpdir-related boilerplate.
Rework all of the tests to use the two managers where appropriate, so
that this:
def test_bloom():
with no_lingering_errors(), test_tempdir('bup-tbloom-') as tmpdir:
...
While making the changes, clean up the imports to match the PEP 8 import
recommendations a bit more closely (excepting the suggestion to avoid
"import bar, baz", which seems odd given the blessing of "from foo
import bar, baz", where bar and baz could still be modules).
Thanks to Tadej Janež for reporting the initial issue.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 26 Mar 2016 23:40:57 +0000 (18:40 -0500)]
test_server_split_with_indexes: close packwriter
Otherwise there are random test failures when the tmpdir is removed
before the PackWriter's destructor calls close(). This doesn't cause an
immediate problem, but it adds an unexpected entry to saved_errors that
causes later tests to fail.
The related saved_errors entries looked like this:
['../../../bup', 'midx', '--auto', '--dir', ...]: returned 15
and the underlying (hidden) midx stderr output in that case looked like
this:
error: '...' is not a bup repository; run "bup init"
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 28 Feb 2016 16:21:57 +0000 (10:21 -0600)]
fuse: remove redundant custom Stat class
It looks like our Stat class was just reinitializing all of the things
that the built in fuse.Stat class already initializes, so just rely on
theirs, and use their keyword initializers to avoid a few more
reinitializations.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Mar 2016 17:45:08 +0000 (11:45 -0600)]
web: fix broken GETs and test
It appears that content GETs, i.e. "curl
http://localhost/foo/latest/bar" may have been broken for a while. To
addresss that, switch to the apparently preferred upstream coroutine
approach, and add some initial tests.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Jan 2016 19:44:04 +0000 (13:44 -0600)]
Never omit explicitly named mounts during index -x
Assuming mounts like this:
/
/usr
and an index invocation like "bup index -x / /usr", don't skip indexing
/usr. Previously bup would skip /usr when -x was specified because
reduce_paths() would drop /usr from the list of paths to index, assuming
that /usr would be covered when indexing /. However, when /usr was
actually reached, -x would cause it to be skipped.
To fix that, exempt all mount points mentioned on the command line from
-x during filesystem traversal, expand the (root only) tests to cover
this situation, and check some corner cases.
Thanks to Wayne Scott for reporting the problem, and Yves-Alexis Perez
for a reminder.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 5 Mar 2016 17:21:50 +0000 (11:21 -0600)]
index: raise children_n limit to struct 'I' max
This assertion appears to be a sanity check, and one that's perhaps too
low (think giant Maildirs), so raise it to the real limit, which should
be UINT_MAX, given that the index header containing it is
encoded/decoded via 'I' (platform native unsigned int):
https://docs.python.org/2/library/struct.html
Since Python doesn't appear to provide access to UINT_MAX, add it (and
INT_MAX) to _helpers.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 21 Dec 2015 19:35:24 +0000 (13:35 -0600)]
index: always return at least the root for filter prefixes
Return at least an entry for the prefix itself for each prefix passed
to filter. Otherwise something like "bup save ... x/y" will fail when
x is up to date because filter will return nothing, save will traverse
nothing in its main loop, and bup will crash with an assertion
failure:
File "/home/rlb/src/bup/main/cmd/bup-save", line 440, in <module>
assert(len(shalists) == 1)
Instead "bup save ... x/y" should (and now will) produce a new save
containing x/y.
Thanks to Simon Persson for helping formulate a good test case.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Paul Kronenwetter <kronenpj@gmail.com>
[rlb@defaultvalue.org: remove everything except preamble update;
rebase on current gc branch; adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Paul Kronenwetter <kronenpj@gmail.com>
[rlb@defaultvalue.org: remove everything except preamble update;
rebase on current rm branch; adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 20 Oct 2014 23:44:06 +0000 (18:44 -0500)]
Add probabilistic "bup gc", but require --unsafe
WARNING: This code is UNSAFE, please test it with care. Among other
things, it adds the first bup command that intends to modify packfiles.
Also, the code assumes/requires a "quiet" repository; any simultaneous
access is dangerous.
Add a "bup gc" command that uses a Bloom filter to track the live
objects during the mark phase. This means that the collection is
probabilistic -- that it may retain some (known) percentage of
garbage, but it can also work within a reasonable, fixed RAM budget
for any particular percentage and repository size.
In many cases, this may be sufficient and preferable, but in the
longer run, we might also want to add a precise method that guarantees
bup retains *no* unreachable/dead objects -- nothing here should
prevent that.
This command is potentially dangerous, so until we've had broader
testing, require all invocations to specify --unsafe, and make it clear
in the documentation that this command isn't considered stable.
Thanks to Tim Riemenschneider for suggesting the pruning of already
visited trees, and an earlier implementation thereof. Thanks to Ben
Kelley for testing and reporting bugs in earlier versions of the
current implementation.
Thanks to Ben Kelly for help devising this Bloom-based approach, and for
testing and reporting bugs in earlier versions.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 29 Dec 2015 20:53:54 +0000 (14:53 -0600)]
Add "bup rm", but require --unsafe invocation
Allow the removal of branches, and the removal of saves from specified
branches. This command only removes the references, so until "bup gc"
is available, all of the related data will still be in the repository,
though possibly difficult to reach, unless otherwise tagged.
This command is potentially dangerous, so until we've had broader
testing, require all invocations to specify --unsafe, and make it clear
in the documentation that this command isn't considered stable.
Thanks to Nix for reporting an earlier mistake in the manpage.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Jan 2016 18:08:56 +0000 (12:08 -0600)]
Sync pack data more carefully when writing
Otherwise a system failure might leave a broken repo. Note that these
changes may not be comprehensive, but they're a start, and hopefully
deal with some of the most critical data.
In addition to syncing the pack data itself, sync the parent directory
after the final rename so that we don't risk losing the reference to
the new file.
Rob Browning [Sat, 9 Jan 2016 18:55:45 +0000 (12:55 -0600)]
helpers: rename realpath to resolve_parent
helpers.realpath does not behave quite like os.path.realpath since it
doesn't resolve any leaf symlink, so to avoid confusion, rename it to
resolve_parent.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 31 Dec 2015 20:55:36 +0000 (14:55 -0600)]
Capture and check top level wvtest watch messages
During wvtest watch, report top level protocol messages to stderr so
that they can be safely captured for examination by report. Continue
printing those message to stdout too, so that "wvtest watch
... 2>test-top.log" will still display them.
Adjust "make check" to include the top level messages when looking for
failures. Otherwise a non-zero exit status from a t/test-*.sh test that
produced no wvtest failure output of its own was just ignored. This can
be demonstrated by adding an "exit 1" to the top of an existing test and
running "make -j check".
Thanks to Greg Troxel for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 30 Dec 2015 19:57:14 +0000 (13:57 -0600)]
Makefile: don't conflate DESTDIR with MANDIR etc.
Don't prepend DESTDIR to the MANDIR, DOCDIR, BINDIR, and LIBDIR values.
Create new dest_* variables instead, so that we don't lose the original
information provided by something like "make MANDIR=... install".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 29 Dec 2015 02:00:05 +0000 (20:00 -0600)]
wvtest.py: set wvtest to wvtest module if __main__
Without this, references to wvtest.* may fail, which can be demonstrated
(for example) by adding a "raise AttributeError('foo')' to
_add_posix1e_acl just before the "if posix1e.has_extended(path)" test.
Then "./wvtest.py lib/bup/t/tgit.py" will fail in the end with:
NameError: global name 'wvtest' is not defined
As a result, wvtest.py never prints a "! ... FAILED" message for the
test, and so "wvtest report" never realizes there was a problem, and
then "make check" exits with zero status.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 17 Dec 2015 23:24:33 +0000 (17:24 -0600)]
Build docs via top Makefile
Move the operations in Documentation/Makefile to ./Makefile, and rework
the process to make it less likely we might overlook command
failures (like ./bup version). Delete Documentation/Makefile.
This may also improve the accuracy of the build dependencies.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 18 Dec 2015 20:59:54 +0000 (14:59 -0600)]
Generate bup-python from Makefile not configure
Don't generate cmd/bup-python in configure. Instead, just define a
bup_python config variable (via config/config.vars.in), and use it to
generate bup-python from ./Makefile.
In support of that, include config/config.vars in ./Makefile, and add
a rule to generate it. This simplifies some of the dependencies, and
removes a level of indirection when determining the correct python to
use (i.e. PYTHON via $(shell)).
This does mean that a "make clean" on a clean tree will run configure.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 18 Dec 2015 23:52:39 +0000 (17:52 -0600)]
"Fix" tests with differing logical/physical paths
Explicitly "cd $(pwd -P)" before testing to work around problems
relating to unresolved questions about the correct behavior of drecurse
when the logical and physical paths differ.
The problem can be reproduced by entering a bup source directory via a
symlink and running "make check":
ln -s bup-real bup
cd bup
make check
Thanks to Brandon Smith and Greg Troxel for reporting the
problem (again) and proposing slightly different solutions.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 15 Dec 2015 01:30:47 +0000 (19:30 -0600)]
Makefile: fix tests with bash and pipefail
The changes[1] to support parallel testing introduced a pipe to tee,
which masks the exit status of a failed test. Fix that by requiring
bash and specifying pipefail.
Rob Browning [Sat, 12 Dec 2015 19:30:18 +0000 (13:30 -0600)]
Configure python, use it, and embed during install
Search for a "reasonable" python in ./configure (or respect PYTHON), and
use that to create a source tree cmd/python-cmd.sh that all of the
relevant invocations in the source tree respect.
During install, rewrite all of the relevant #! lines to refer to the
./configure selected Python.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
David Kettler [Mon, 30 Nov 2015 13:28:28 +0000 (23:58 +1030)]
test-tz.sh: test save in a fractional time zone
Prior to this change[1], the commit time zone offset was generated
incorrectly for non-integer hour offsets from UTC; seconds, rather
than minutes, were output. Such values in a bup commit cause
subsequent attempts to access the repository to fail with "Exception:
cannot parse commit".
When run with TZ=Australia/Adelaide the test suite failed in several
places. Add a test which sets the time zone explicitly and tests a
save for consistency.
Rob Browning [Sat, 5 Dec 2015 19:41:36 +0000 (13:41 -0600)]
Fix utc_offset_str for fractional TZs
Previously, for fractional offsets like TZ=FOO+3:3 (where the offset
is not an integer number of hours), utc_offset_str() would return a
large, incorrect integer value. Adjust it to handle arbitrary
offsets, and document that it truncates any fractional seconds of the
offset.
Before this fix, whenever bup was operating with a fractional offset
(e.g. TZ=Australia/Adelaide), it wrote commits (i.e. saves) that it
won't be able to read. Attempts to read them will fail with
"Exception: cannot parse commit".
Thanks to David Kettler for reporting the problem, and proposing an
alternate fix.
Reviewed-by: David Kettler <kettler@internode.on.net> Tested-by: David Kettler <kettler@internode.on.net>
[rlb@defaultvalue.org: changed / to // for Python 3 compat] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 5 Dec 2015 19:10:45 +0000 (13:10 -0600)]
Call tzset before localtime_r as required by POSIX
Otherwise localtime_r() may not respect the current TZ setting.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: David Kettler <kettler@internode.on.net>
projects / bup.git / log