Signed-off-by: Paul Kronenwetter <kronenpj@gmail.com>
[rlb@defaultvalue.org: remove everything except preamble update;
rebase on current rm branch; adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 29 Dec 2015 20:53:54 +0000 (14:53 -0600)]
Add "bup rm", but require --unsafe invocation
Allow the removal of branches, and the removal of saves from specified
branches. This command only removes the references, so until "bup gc"
is available, all of the related data will still be in the repository,
though possibly difficult to reach, unless otherwise tagged.
This command is potentially dangerous, so until we've had broader
testing, require all invocations to specify --unsafe, and make it clear
in the documentation that this command isn't considered stable.
Thanks to Nix for reporting an earlier mistake in the manpage.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Jan 2016 18:08:56 +0000 (12:08 -0600)]
Sync pack data more carefully when writing
Otherwise a system failure might leave a broken repo. Note that these
changes may not be comprehensive, but they're a start, and hopefully
deal with some of the most critical data.
In addition to syncing the pack data itself, sync the parent directory
after the final rename so that we don't risk losing the reference to
the new file.
Rob Browning [Sat, 9 Jan 2016 18:55:45 +0000 (12:55 -0600)]
helpers: rename realpath to resolve_parent
helpers.realpath does not behave quite like os.path.realpath since it
doesn't resolve any leaf symlink, so to avoid confusion, rename it to
resolve_parent.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 31 Dec 2015 20:55:36 +0000 (14:55 -0600)]
Capture and check top level wvtest watch messages
During wvtest watch, report top level protocol messages to stderr so
that they can be safely captured for examination by report. Continue
printing those message to stdout too, so that "wvtest watch
... 2>test-top.log" will still display them.
Adjust "make check" to include the top level messages when looking for
failures. Otherwise a non-zero exit status from a t/test-*.sh test that
produced no wvtest failure output of its own was just ignored. This can
be demonstrated by adding an "exit 1" to the top of an existing test and
running "make -j check".
Thanks to Greg Troxel for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 30 Dec 2015 19:57:14 +0000 (13:57 -0600)]
Makefile: don't conflate DESTDIR with MANDIR etc.
Don't prepend DESTDIR to the MANDIR, DOCDIR, BINDIR, and LIBDIR values.
Create new dest_* variables instead, so that we don't lose the original
information provided by something like "make MANDIR=... install".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 29 Dec 2015 02:00:05 +0000 (20:00 -0600)]
wvtest.py: set wvtest to wvtest module if __main__
Without this, references to wvtest.* may fail, which can be demonstrated
(for example) by adding a "raise AttributeError('foo')' to
_add_posix1e_acl just before the "if posix1e.has_extended(path)" test.
Then "./wvtest.py lib/bup/t/tgit.py" will fail in the end with:
NameError: global name 'wvtest' is not defined
As a result, wvtest.py never prints a "! ... FAILED" message for the
test, and so "wvtest report" never realizes there was a problem, and
then "make check" exits with zero status.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 17 Dec 2015 23:24:33 +0000 (17:24 -0600)]
Build docs via top Makefile
Move the operations in Documentation/Makefile to ./Makefile, and rework
the process to make it less likely we might overlook command
failures (like ./bup version). Delete Documentation/Makefile.
This may also improve the accuracy of the build dependencies.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 18 Dec 2015 20:59:54 +0000 (14:59 -0600)]
Generate bup-python from Makefile not configure
Don't generate cmd/bup-python in configure. Instead, just define a
bup_python config variable (via config/config.vars.in), and use it to
generate bup-python from ./Makefile.
In support of that, include config/config.vars in ./Makefile, and add
a rule to generate it. This simplifies some of the dependencies, and
removes a level of indirection when determining the correct python to
use (i.e. PYTHON via $(shell)).
This does mean that a "make clean" on a clean tree will run configure.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 18 Dec 2015 23:52:39 +0000 (17:52 -0600)]
"Fix" tests with differing logical/physical paths
Explicitly "cd $(pwd -P)" before testing to work around problems
relating to unresolved questions about the correct behavior of drecurse
when the logical and physical paths differ.
The problem can be reproduced by entering a bup source directory via a
symlink and running "make check":
ln -s bup-real bup
cd bup
make check
Thanks to Brandon Smith and Greg Troxel for reporting the
problem (again) and proposing slightly different solutions.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 15 Dec 2015 01:30:47 +0000 (19:30 -0600)]
Makefile: fix tests with bash and pipefail
The changes[1] to support parallel testing introduced a pipe to tee,
which masks the exit status of a failed test. Fix that by requiring
bash and specifying pipefail.
Rob Browning [Sat, 12 Dec 2015 19:30:18 +0000 (13:30 -0600)]
Configure python, use it, and embed during install
Search for a "reasonable" python in ./configure (or respect PYTHON), and
use that to create a source tree cmd/python-cmd.sh that all of the
relevant invocations in the source tree respect.
During install, rewrite all of the relevant #! lines to refer to the
./configure selected Python.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
David Kettler [Mon, 30 Nov 2015 13:28:28 +0000 (23:58 +1030)]
test-tz.sh: test save in a fractional time zone
Prior to this change[1], the commit time zone offset was generated
incorrectly for non-integer hour offsets from UTC; seconds, rather
than minutes, were output. Such values in a bup commit cause
subsequent attempts to access the repository to fail with "Exception:
cannot parse commit".
When run with TZ=Australia/Adelaide the test suite failed in several
places. Add a test which sets the time zone explicitly and tests a
save for consistency.
Rob Browning [Sat, 5 Dec 2015 19:41:36 +0000 (13:41 -0600)]
Fix utc_offset_str for fractional TZs
Previously, for fractional offsets like TZ=FOO+3:3 (where the offset
is not an integer number of hours), utc_offset_str() would return a
large, incorrect integer value. Adjust it to handle arbitrary
offsets, and document that it truncates any fractional seconds of the
offset.
Before this fix, whenever bup was operating with a fractional offset
(e.g. TZ=Australia/Adelaide), it wrote commits (i.e. saves) that it
won't be able to read. Attempts to read them will fail with
"Exception: cannot parse commit".
Thanks to David Kettler for reporting the problem, and proposing an
alternate fix.
Reviewed-by: David Kettler <kettler@internode.on.net> Tested-by: David Kettler <kettler@internode.on.net>
[rlb@defaultvalue.org: changed / to // for Python 3 compat] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 5 Dec 2015 19:10:45 +0000 (13:10 -0600)]
Call tzset before localtime_r as required by POSIX
Otherwise localtime_r() may not respect the current TZ setting.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: David Kettler <kettler@internode.on.net>
Rob Browning [Sun, 29 Nov 2015 18:38:03 +0000 (12:38 -0600)]
Choose 2M when SC_ARG_MAX is -1 (unspecified)
When sysconf() returns -1 for a limit, that means the limit is
"unspecified". In that case, choose 2M for SC_ARG_MAX instead of the
POSIX minimum, matching (somewhat arbitrarily) the value on a current
16GB Debian amd64 system.
The POSIX minimum was unnecessary, likely inefficient, and causing
unnecessary failures.
Thanks to Mark J Hewitt <m.hewitt@computer.org> for reporting the
problem and tracking down the cause.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 Jul 2015 15:26:35 +0000 (10:26 -0500)]
vfs: remove support for /foo/TAG (keep /.tag/TAG)
Don't interleave the tags with the branch save dates in the VFS. Doing
so meant that any access to the /branch required retrieving all tags in
the repository. It also introduced the possibility of duplicate tag and
date names.
Tags are still available in the global /.tag directory.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 3 Jul 2015 17:34:24 +0000 (12:34 -0500)]
Handle mincore cross-platform differences
On the BSDs, mincore is defined to return char values rather than
unsigned char values, so test for that and adjust our invocation.
Also check for MINCORE_INCORE, which is used on some platforms to test
the returned status bytes. When it's found, define
helpers.MINCORE_INCORE appropriately.
Rework the mincore-related code, moving much of the fmincore C code to
Python to avoid dealing with platform-related strerror_r differences,
and replace the _helpers fmincore with a more primitive mincore.
To accommodate the more complicated ./configure testing, require bash,
and use printf -v to (safely) set ac_defined_$name during all calls to
AC_DEFINE so that we can use the discovered values to guard tests during
configuration.
Thanks to Thomas Klausner for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 14 Jul 2015 00:33:24 +0000 (19:33 -0500)]
Handle sysconf results more carefully
Sysconf indicates that there's no definite limit by returning -1 and
leaving errno unchanged. In that case, for SC_ARG_MAX, guess 4096. For
SC_PAGE_SIZE, die, since various operations currently require a
page_size.
Thanks to Mark J Hewitt for reporting the issue, and to Mark and Nix for
investigating the cause.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 30 May 2015 16:18:26 +0000 (11:18 -0500)]
Eject our pages after save via fmincore
Use fmincore and fadvise_done to eject only pages that have become
resident during our save traversal (i.e. via hashsplitting) in
batches (currently 8MB or one VM page, whichever's larger).
Hopefully this will work better than either universal ejection (our
previous behavior), or no ejection (which dramatically slows down save
operations on some systems, perhaps due to competition with access to
the indexes, etc.
Thanks to Nimen Nachname for the initial suggestion, and thanks to Tilo
Schwarz for reporting bugs in a previous version of the patch, and for
noting that we shouldn't wait until the end of a large region before
starting to eject it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Ben Wiederhake <Ben.Wiederhake@gmail.com> Tested-by: Tilo Schwarz <mail@tilo-schwarz.de>
[rlb@defaultvalue.org: bup_fmincore: add missing malloc result check,
and missing free(result) when munmap fails to 437fedd07ee327c14b11cb19f6c0519ef1e50884] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 26 May 2015 01:55:32 +0000 (20:55 -0500)]
Revert "Avoid fadvise (since it doesn't work as expected)"
On some systems (the reported system had 2GB), not forcing out all of
the data traversed during save dramatically slows down save operations,
possibly due to competition with access to the indexes, etc. So restore
the use of fadvise_done() for now.
Rob Browning [Sun, 14 Jun 2015 14:58:32 +0000 (09:58 -0500)]
test.sh: separate index and split/join tests
Move the index tests to test-index.sh and the split/join tests to
test-split-join.sh in order to increase the potential parallelism (and
modularity/isolation).
Here, this brings the "make -j check" time down quite a bit. Before
test.sh was always the last thing running, and the index tests took most
of the time.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 13 Jun 2015 17:54:21 +0000 (12:54 -0500)]
test-fuse: format save name with python, not bash
Some versions of bash don't support the date expansion we used.
Thanks to pspdevel for reporting the issue.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Gabriel Filion <gabster@lelutin.ca> Tested-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: add comment above savename()] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Ben Kelly [Thu, 4 Jun 2015 16:47:27 +0000 (12:47 -0400)]
bup midx: fix --output when used with --auto or --force
This fixes an issue where --output is properly respected only when
neither of these options are used.
Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: rebased onto e25363fc58cd906337ecee28d715af8f355fd921] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 23 May 2015 16:30:36 +0000 (11:30 -0500)]
Update definition lists for pandoc 1.13
It appears that pandoc 1.13 requires a blank line between definition
list items, so add it. See "compact_definition_lists" in the pandoc
README for more information.
Thanks to Robert Edmonds for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 May 2015 21:33:16 +0000 (16:33 -0500)]
Avoid fadvise (since it doesn't work as expected)
Currently, it appears that at least on Linux posix_fadvise() always
purges the cache given POSIX_FADV_DONTNEED, which is what bup uses, and
does nothing for POSIX_FADV_NOREUSE, which means that before this patch,
a bup save would completely clear the filesystem cache of any file data
traversed during the run. Aside from being completely unintended, this
meant that active VM images, large databases, etc. would probably be
purged with every save.
Since it also looks like at least NetBSD doesn't do anything for
DONTNEED, and tools like tar, cpio, and duplicity don't use it, let's
just drop it. That's simpler, and we can always add it back if/when
someone discovers it helps somewhere relevant.
Thanks to Nimen Nacnamme for reporting the issue.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org> Acked-by: Greg Troxel <gdt@lexort.com>
Update the example of making a backup to a remote server in README.md.
Replace 'ssh SERVENAME bup init' with 'bup init -r SERVERNAME'. The
latter doesn't only initialize the remote repository, but also the local
one (if it doesn't exist). Augment 'bup {init,save} -r SERVERNAME'
commands with the path specifier to show the ability to specify the
remote path.
Signed-off-by: Tadej Janež <tadej.j@nez.si> Reviewed-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: shorten/adjust commit summary; adjust and change
tense of commit message.] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 23 Apr 2015 20:41:45 +0000 (22:41 +0200)]
Reject invalid string in --date argument
As parse_date_or_fatal() currently uses atof(), which just returns 0
if the string isn't a valid number, it can never actually be fatal
and will just use "1970-01-01 00:00:00" as the time if the string is
specified wrong.
Fix that by using float() directly so ValueError() is raised.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 28 Mar 2015 20:09:47 +0000 (15:09 -0500)]
Get TZ offset from C localtime, given tm_gmtoff
If we detect that struct tm contains tm_gmtoff, use the system
localtime() to compute timezone offsets. This may help fix problems on
platforms where Python strftime "%z" doesn't report accurate timzeone
information.
Thanks to Patrick Rouleau for reporting just such a problem on Cygwin.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 28 Mar 2015 17:48:19 +0000 (12:48 -0500)]
Adjust sparse restore tests for test fs block size
Change test-sparse-files.sh to detect the test fs block size (when
possible) and adjust its behavior accordingly. If the block size can't
be determined, use a block size of 3MB, which is hoped to be larger than
any block sizes we'll encounter anytime soon.
Previously the tests might fail on filesystems with relatively large
block sizes, like those on the current Debian powerpc and ppc64el build
daemons (64k).
Thanks to Goswin Brederlow for mentioning that the Lucene block size is
1MB, to Robert Edmonds for running a build through the Debian buildds,
which revealed the problem, and to Julien Cristau for reporting the
block size on the failing buildds.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Mar 2015 21:43:47 +0000 (16:43 -0500)]
Rework write_sparsely() to fix in-buffer zero runs
Fix the sparse restoration of buffers that have non-zero bytes, followed
by a run of zero bytes that's longer than the minimum sparse run
length (currently 512), followed by non-zero bytes.
Previously, the initial non-zero bytes would be *lost*.
In the new code, don't unconditionally output previous zero bytes --
merge them with any leading zeros in the current block.
And allow arbitrarily large sparse regions; use append_sparse_region()
to break up runs that are too large for off_t into a sequence of seeks
of no more than INT_MAX bytes each.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Mar 2015 21:38:34 +0000 (16:38 -0500)]
Test sparse restore of short in-buffer zero run
Test that sparse --restore handles the case where within one call to
write_sparsely() (one buffer) we have non-zero bytes, followed by a run
of zero bytes that's longer than the minimum sparse run
length (currently 512), followed by non-zero bytes.
Currently, the initial non-zero bytes will be lost, and this test will
fail.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log