Gabriel Filion [Tue, 14 May 2013 06:53:05 +0000 (02:53 -0400)]
Documentation: replace ~/.bup with $BUP_DIR in bup-index.md and bup-save.md.
When one is not using the default bupdir placement, then those files
aren't found inside ~/.bup, but wherever "-d /path.." or $BUP_DIR are
pointing to.
Thanks to krichter722@aol.de for mentioning this.
Signed-off-by: Gabriel Filion <gabster@lelutin.ca> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Fix a timing issue that often caused the test to fail -- the (removed)
file "foo" was marked as invalid in the index when it shouldn't have
been because it was modified less than tmax before the index run.
Insert a "bup tick" just before "bup index" so that foo won't be
marked invalid by the second run.
Don't expect the second "bup save" to fail since foo will now be valid
in the index (and hence ignored), even though it is no longer
available. The removal will be noticed by the next index+save.
When removing directories, the outcome is different: since metadata is
saved per directory, "bup save" notices that the directory is missing
and logs an "errno 2" error, but does not crash, which is what the
tested patch is concerned with. A third index+save run completes
without error.
This also means that without a "bup tick" before an index run, calls,
"bup save" might exit with non-zero exitcode when files/directories
are removed after bup index, but bup is not guaranteed to notice
missing files.
Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Add tests for file and dir removal between index and save.
Make sure the removals don't cause bup save to immediately abort.
Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
[rlb@defaultvalue.org: move tests to their own file.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Damien Robert [Tue, 20 Nov 2012 20:52:45 +0000 (21:52 +0100)]
save-cmd.py: don't crash when a path disappears between index and save.
Protect calls to metadata.from_path() with a try/catch and defer
errors via add_error() instead of just crashing.
Signed-off-by: Damien Robert <damien.olivier.robert@gmail.com>
[git@tim-riemenschneider.de: rebase to current tmp/pending/meta: since
metadata is (now) stored in the index, only 1 of 2 hunks still
applies (race condition between reading the file for its content and
stat-ing it for metadata).]
[rlb@defaultvalue.org: edit commit message; squash file and dir
removal fixes into this commit; limit the scope of the try/catch to
the from_path() call, and put the remaining code in an else clause.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 18 Aug 2012 20:35:23 +0000 (15:35 -0500)]
Add support for "bup restore --exclude-rx <pattern> ...".
When --exclude-rx <pattern> is provided to bup restore, don't restore
any path matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html). The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards". To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".
You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.
Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 21 Jul 2012 20:09:47 +0000 (15:09 -0500)]
Add support for "bup index --exclude-rx <pattern> ...".
When --exclude-rx <pattern> is provided to bup index, exclude any path
matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html). The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards". To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".
You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.
Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Thomas Haller [Sun, 3 Mar 2013 19:31:57 +0000 (20:31 +0100)]
wvtest.py: add a fallback definition of relpath() for Python < 2.6.
Since bup still supports Python 2.5 and relpath was added to Python in
2.6, add an os.path.relpath replacement, adapted from CPython source
(tag v2.6, file Lib/posixpath.py, hg-commit 95fff5a6a276).
Signed-off-by: Thomas Haller <thom311@gmail.com>
[rlb@defaultvalue.org: update LICENSE and add comment to code about
updating LICENSE.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Zak Wilcox <iwilcox@iwilcox.me.uk>
[rlb@defaultvalue.org: change original --ignore-dev to --no-check-device to match tar.
Adjust code to work with current master.
Remove tests -- will be reintroduced shortly.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Sun, 17 Mar 2013 22:24:08 +0000 (18:24 -0400)]
Ignore artefacts from sampledata
9b5ae5f53e introduced the creation of some symlinks to avoid issues on
Cygwin. "make" now leaves the repository in a dirty state since we
forgot to exclude those symlinks.
The symlinks in question are products of the build/test process so we
need to avoid commiting them.
Signed-off-by: Gabriel Filion <gabster@lelutin.ca> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Jakob Matthes [Sun, 10 Mar 2013 18:10:38 +0000 (19:10 +0100)]
bup drecurse/index: allow non-directory --exclude and --exclude-from paths.
Previously bup only allowed directory exclusions.
Signed-off-by: Jakob Matthes <jakob.matthes@gmail.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com>
[rlb@defaultvalue.org: change test file from g to j to avoid conflict
with subsequent "exclude-from" test; add WVPASS to mkdir so problem
is evident; edit commit message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Mark J Hewitt [Mon, 1 Oct 2012 16:17:28 +0000 (17:17 +0100)]
Use the correct C types when converting between C and Python in _helpers.c.
Previously, bup might fail when handling large repositories. For
example:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-midx", line 258, in <module>
do_midx_dir(path)
File "/usr/lib/bup/cmd/bup-midx", line 199, in do_midx_dir
all = list(do_midx_group(path, part1)) + part2
File "/usr/lib/bup/cmd/bup-midx", line 216, in do_midx_group
rv = _do_midx(path, None, sublist, gprefix)
File "/usr/lib/bup/cmd/bup-midx", line 123, in _do_midx
count = merge_into(fmap, bits, total, inp)
OverflowError: size does not fit in an int
['/usr/bin/bup', 'midx', '--auto', '--dir',
'/backup/bup/repository/objects/pack']: returned 1
Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: Remove vestigal header includes.
Remove Py_ssize_t definition for Python < 2.5.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
save-cmd.py: insist on having an index; check for the primary index file.
Insist on having an index, and report an error (with a suggestion to
run "bup index") if one isn't found.
Previously (at least as recently as 0.25-rc1), bup save would just
create an empty tree if there was no index. The metadata work
unintentionally broke that behavior.
For now, treat a missing index as an error under the assumption that
it's not likely to be intentional. If a good use case for the
previous behavior is presented, we can always restore it.
Signed-off-by: Jean-Baptiste Denis <jbd@jbdenis.net>
[rlb@defaultvalue.org: adjust commit message and save-cmd.py error message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 17 Feb 2013 16:33:08 +0000 (10:33 -0600)]
Normalize stat(2) timespecs in _helpers.c.
Previously, the code required timespecs to have a nanosecond value
between 0 and 999999999, because that's what Linux appeared to
produce, but other platforms don't (i.e. Cygwin) -- so normalize the
system values to match expectations.
In particular, look for negative nanosecond values (must be greater
than -10**9), and if found, rewrite the timespec with non-negative
nanoseconds (still less than 10**9).
Thanks to Will Rouesnel <w.rouesnel@gmail.com> for reporting the
problem and posting an initial patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Ryan Brown [Thu, 7 Mar 2013 02:48:23 +0000 (21:48 -0500)]
Add a force-delete function to t/test.sh
Replace "rm -rf" with force-delete because "rm -rf" will proceed even
if it doesn't succeed. This masked a problem (reported on Fedora)
where the root directory and other system directories had "r-xr-xr-x"
(0555) permissions, causing bup restore to create directories that "rm
-rf" couldn't remove. As a result, subsequent tests would fail.
Signed-off-by: Ryan Brown <ryansb@csh.rit.edu>
[rlb@defaultvalue.org: edited commit message.] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Mar 2013 22:51:14 +0000 (16:51 -0600)]
Fix use of Python assert() with respect to optimization.
Fix a number of places where bup's assertions had material
side-effects, or where other code expected to see the AssertionError,
neither of which happen when optimization is enabled.
Reported-by: Jon Dowland <jmtd@debian.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 10 Nov 2012 19:06:33 +0000 (13:06 -0600)]
Change index; include atime; pack time as xstat timespec; use ns in memory.
Update the index format header to 'BUPI\0\0\0\4' (version 4).
Change the index to include atimes, and write all times to bupindex as
xstat timespecs. This is in preparation for indexing all metadata.
After moving all of the times to the index, there should be far fewer
unique instances of the remaining metadata in many/most cases.
Change the index and index-related code to handle all time
values as integer nanoseconds since the epoch, excepting the packed
format, which (as mentioned above) is now an xstat timespec.
After these changes the index-related in-memory and on-disk time value
handling should match that of the metadata code (i.e. metadata.py,
.bupm, etc.).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: change 10e8 to 10**9 for ns per second.]
Rob Browning [Sun, 23 Sep 2012 23:26:48 +0000 (18:26 -0500)]
Cache all password (pwd) and group (grp) database lookups.
Thanks to Jann Horn <jannhorn@googlemail.com> for determining that
even just caching user names and group names could provide a notable
performance improvement in some cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 18 Aug 2012 20:26:28 +0000 (15:26 -0500)]
Overhaul restore destination handling, and stripping/grafting behavior.
Change restore to respond to source paths like this (assume outdir
corresponds to "." if there no -C argument or to -C outdir):
/foo/what/ever - extract ever to outdir/ever
/foo/what/ever/ - extract ever/* to outdir/*
/foo/what/ever/. - extract ever/. to outdir/. (i.e. outdir == ever).
Also fix handling of top-level commit symlinks. Previously bup would
just restore /foo/latest as a dummy symlink like this:
latest -> ../.commit/SHA
Instead, dereference latest and restore the target instead.
Tighten up stripping/grafting with additional argument checks, and
handle any root collisions by creating a fake root dir (see comments
in save-cmd.py). Bup still doesn't handle other path collisions yet,
i.e. if both /foo/bar and /bar are remapped to /bar.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 5 May 2012 17:45:01 +0000 (12:45 -0500)]
Add preliminary hardlink support for review.
Thanks to Tim Riemenschneider <git@tim-riemenschneider.de> for
pointing out a compatibility problem (with older versions of Python)
in an earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Zoran Zaric [Sat, 26 Jan 2013 18:44:34 +0000 (19:44 +0100)]
Print a more meaningful error message when help can't run man.
Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: elaborated on the error message a bit] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Alexander Barton [Sat, 26 Jan 2013 18:48:47 +0000 (19:48 +0100)]
Explicitly use "du -k" to set block size in t/test.sh.
Commit e3514812 (Explicitly select the "du" block size in t/test.sh)
used the parameter "-B 1024" which isn't portable. Now use "du -k"
to explicitly get output in kilobytes.
At least du(1) on Linux (GNU coreutils), Mac OS X, FreeBSD, NetBSD,
and OpenBSD support this option.
Signed-off-by: Alexander Barton <alex@barton.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 23 Jan 2013 01:19:20 +0000 (19:19 -0600)]
Explicitly select the "du" block size in t/test.sh.
Since du's default output can be affected by several environment
variables (DU_BLOCK_SIZE, BLOCK_SIZE and BLOCKSIZE), be explicit about
what we want via -B 1024.
Thanks to Michael Ekstrand <michael@elehack.net> for the original
report, and help with the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Yung-Chin Oei [Wed, 24 Oct 2012 02:26:35 +0000 (03:26 +0100)]
Remove dependency on linux/ext2_fs.h
Due to a missing header in (some versions of) linux/ext2_fs.h,
configure's AC_CHECK_HEADERS test fails for this header, making it
appear "not found", and causing Linux-attr support to be disabled in bup
for no good reason.
Given that all the flags bup needs are available in linux/fs.h, we can
safely remove the requirement for linux/ext2_fs.h altogether, to work
around the problem.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Jan 2013 16:23:04 +0000 (10:23 -0600)]
Check the arguments passed to ./configure.
For now, don't allow arguments, since ./configure doesn't actually
support any. Previously "./configure --help" would print misleading
information, and any other arguments were silently ignored.
Thanks to Alexander Barton <alex@barton.de> for the report.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 21 Jul 2012 20:26:17 +0000 (15:26 -0500)]
Add a metadata() method to Node, File, Symlink, and Dir VFS objects.
The current implementation just populates the self._metadata field of
all of the objects in a directory (and of directory itself) from the
relevant .bupm file whenever metadata() is called on any one of those
objects.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Reviewed-by: Gabriel Filion <lelutin@gmail.com>
t/test.sh: fix test failure when $top_dir has a dot-name
Fix the admittedly rare case where we're running tests from a tree
that has a dot-name at its base, which only 'bup ls -a' will print.
This problem was originally encountered on a system that keeps mounts
under '/.autofs/'.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 1 Oct 2011 17:24:36 +0000 (12:24 -0500)]
Save metadata during "bup save".
Record metadata in a hidden/mangled file named .bupm in each
directory, so that the metadata for /foo/ is stored in /foo/.bupm,
with the first entry being the metadata for foo/ itself.
Record an empty index file for each special file so that index entries
and .bupm entries correspond correctly.
Rework the strip/graft functions to return both the save-name, and the
underlying filesystem path to the save-name (when there is one) for
each component. There may not be corresponding filesystem paths if
graft options rewrite the path prefix.
For now, record "/" metadata only when there are no strip/graft
options.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Tested-by: Alexander Barton <alex@barton.de>
Rob Browning [Fri, 23 Nov 2012 20:32:39 +0000 (14:32 -0600)]
Since 'latest' is computed by revs[0], it must exist, so don't test it.
Remove the unnecessary "if latest:" conditional from
BranchList._mksubs(). Given the fact that latest used to be computed
via max(revs), and is now computed by revs[0], there's no way it
wouldn't exist.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 23 Nov 2012 20:18:52 +0000 (14:18 -0600)]
Use branch tip, not newest date to choose 'latest' in BranchList._mksubs().
Previously 'latest' was chosen in the VFS by taking max(revs), which
would pick the most recent commit by rev_list date, which only has a
resolution of integer seconds.
That approach could fail if two commits (i.e. two "bup save" runs)
executed close enough together that both commits ended up with the
same rev_list date. In that case, the sort order was determined by
the commit hash, which is effectively random.
Since I couldn't think of a case where you wouldn't want 'latest' to
refer to the most recent commit (save) to the relevant branch,
'latest' is now computed as revs[0], i.e. the first commit returned by
rev_list.
Thanks to Zoran and Gabriel for some helpful sanity checks while
tracking this down.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log