itxx00 [Wed, 8 Jan 2014 05:14:26 +0000 (13:14 +0800)]
bup-restore.md: always initialize root_meta in do_root.
Previously, using bup split with bup restore would cause an error:
UnboundLocalError: local variable 'root_meta' referenced before assignment
Signed-off-by: itxx00 <itxx00@gmail.com>
[rlb@defaultvalue.org: adjust commit message. Initialize root_meta
unconditionally before guard to match other code.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 3 Dec 2013 18:37:53 +0000 (12:37 -0600)]
Check for overflow when converting from Python to unsigned integer types.
Python's PyArg_ParseTuple() unsigned integer type conversions don't
check for overflow, so stop using them. Add bup_uint_from_py(),
bup_ulong_from_py(), and bup_ullong_from_py() functions that do check,
and use them instead.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 19 Dec 2013 04:16:44 +0000 (22:16 -0600)]
Don't require non-negative timespec ns; fix stat timestamp conversions.
Previously we required that the system timespec tv_nsec values be
non-negative -- stop that. POSIX doesn't specify, and there's no real
need for us to care, since all significant timestamp manipulations are
now handled as integer nanosecond values, not (sec, ns) pairs.
Change stat_struct_to_py() (used by bup_stat()/bup_lstat()) to use
INTEGER_TO_PY() to convert the timestamp st_atime/st_mtime/st_ctime
values so that we don't have to care about the sign/size of the
underlying type (which should be time_t).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 03:12:06 +0000 (21:12 -0600)]
Always publish (l)utimes in helpers when available and fix type conversions.
Previously, bup assumed a timeval was (long, long), which agrees with
glibc.info and (Linux) utimes(2), but POSIX says it's (time_t,
suseconds_t).
Since the types vary (and even if they didn't, Python doesn't provide
safe "time_t" conversions), handle the type conversions more
generically, via ASSIGN_PYLONG_TO_INTEGRAL() and
INTEGRAL_ASSIGNMENT_FITS().
Optimistically assume the latter is sufficient for tv_usec (where we
convert it to a long long) because POSIX guarantees that tv_usec will
be a signed integral.
Whenever available, publish utimes and lutimes independently in
helpers, and run the relevant t/txstat.py tests. So now, we'll be
running at least some tests on utimensat, lutimes, and utimes whenever
they're available.
A more significant improvement would be to run some broader bup tests
twice, once with utimensat and once with utimes/lutimes, whenever both
are available.
Rob Browning [Sat, 28 Dec 2013 18:18:51 +0000 (12:18 -0600)]
Remove vestigial and inappropriate isnan()/isinf() timestamp tests.
I'm fairly sure this is an historical accident, but regardless, remove
the (my) obviously inappropriate floating point tests on integer
values from bup_parse_xutime_args(). Nothing to see here -- move
along.
I've broken this change out from a larger post-0.25 patch, where I'd
already fixed it, so that it'll be easy to pull into stable if we
like.
Thanks to Ed Maste <carpeddiem@gmail.com> for reporting the problem
(which prompted me to go ahead and create the separate fix).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 Dec 2013 21:34:44 +0000 (15:34 -0600)]
Add -n, -A, -F, --file-type, --numeric-ids and detailed -l to "bup ls".
With the addition of -A, change -a to behave more conventionally,
i.e. to show "." and "..".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Alexander Barton <alex@barton.de> Reviewed-By: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: fall back to n.mode when n.metadata() isn't
available in node_info() so that CommitDirs ("ls -AF /"), etc. will
still work; thanks to Gabriel Filion <gabster@lelutin.ca> for the
report; fix the rdiff-backup test to use -A.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 02:56:52 +0000 (20:56 -0600)]
Always publish utimensat in helpers when available and fix type conversions.
Previously, bup assumed a timespec was (long, long), which agrees with
glibc.info, but utimensat(2) and POSIX say it's (time_t, long).
Since the types vary (and even if they didn't, Python doesn't provide
safe "time_t" conversions), add ASSIGN_PYLONG_TO_INTEGRAL(), which
should handle safely assigning a PyLong value to any given C integral
destination, whenever it will fit -- then use that macro to handle the
utimensat arguments.
Whenever it's available, publish utimensat independently in helpers.
For now, this allows us to test utimensat directly, and in the future,
it may allow us to test bup's behavior with either utimensat or
utimes/lutimes without recompilation, whenever both are available.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 02:19:44 +0000 (20:19 -0600)]
Don't fail tests when the timestamp read resolution is higher than write.
Previously, if bup was able to read path timestamps at a higher
resolution than it could write them, tests would fail. This situation
can occur (for example) when the stat() resolution is 1ns, but either
the underlying filesystem's isn't, or bup wasn't able to find
utimensat at build time.
To fix this, compute the maximum resolution of the test filesystem
(via "t/ns-timestamp-resolutions HERE"), and then limit the "bup
xstat" timestamp resolution in the tests to match via a new
--mtime-resolution argument.
For completeness, also add --atime-resolution and --ctime-resolution
arguments.
Thanks to Tim Riemenschneider <t.riemenschneider@detco.de> for
reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 19 Dec 2013 02:19:19 +0000 (20:19 -0600)]
Stop testing negative timestamps -- they're not necessarily valid.
Both timevals and timespecs use a time_t for the seconds which POSIX
requires to be an "integer or real-floating type", so unsigned integer
types are perfectly valid. (See FUSE for example -- though it's not
even using a time_t; at the moment it's just using uint64_t...).
Thanks to Tim Riemenschneider <t.riemenschneider@detco.de> for
reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 16 Dec 2013 02:30:46 +0000 (20:30 -0600)]
Require utimensat or utimes/lutimes.
This was already true (tests would fail), so make it explicit, and if
we do encounter any platforms where none of these functions are
available, we can worry about it then.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Dec 2013 16:18:37 +0000 (10:18 -0600)]
t/test-meta.sh: handle systems that have sgid directories.
Explicitly chown the test directory to a group we're in so that the
tests will work on systems that have sgid directories by default
(i.e. NetBSD, etc.).
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 3 Dec 2013 20:17:26 +0000 (14:17 -0600)]
Create t/force-delete from t/lib.sh force-delete and use it everywhere.
Move the t/lib.sh force-delete code to t/force-delete as a standalone
command, and use it everywhere -- particularly to delete t/tmp during
"make clean", given the recent move to run all tests inside t/ via
TMPDIR (18f838c9a66d521f9fa042eead3c5771ca99b03a).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 25 Nov 2013 20:31:28 +0000 (14:31 -0600)]
Don't check ACLs in t/compare-trees on NetBSD.
Treat NetBSD like Cygwin and Darwin (OS X) in t/compare-trees. This
should finish the process of disabling POSIX ACL support on NetBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Nov 2013 18:05:00 +0000 (12:05 -0600)]
Don't check ACLs in t/compare-trees on FreeBSD.
Treat FreeBSD like Cygwin and Darwin (OS X) in t/compare-trees. This
should finish the process of disabling POSIX ACL support on FreeBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 20 Nov 2013 20:57:53 +0000 (14:57 -0600)]
Enforce MAX_PER_TREE by always _squish()ing in spit_to_shalist().
Previously bup would ignore MAX_PER_TREE whenever it hit a long run of
data that didn't produce a non-zero hashplit "level" (see the
discussion of fanout in DESIGN). That can happen, for example, when
traversing a file containing large zero regions (sparse or not).
As a result, bup could produce an arbitrarily large number of blobs at
level 0 of the hashsplit tree, causing it to consume increasing memory
during split/save, and to behave badly during join/restore.
To fix that, don't try to outsmart _squish() -- just call it every
time, and let it enforce MAX_PER_TREE when appropriate.
Thanks to trebor <robert.rebstock@tempelhof-projekt.de> for reporting
the problem a long while back, Yung-Chin Oei <yungchin@yungchin.nl>
for tracking down the cause and proposing a slightly different fix,
and Aidan Hobson Sayers <aidanphs@gmail.com> for suggesting this
particular approach.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 12 Nov 2013 17:38:35 +0000 (11:38 -0600)]
Don't crash; defer an error when a path changes while indexing it.
Mirror the error handling already used by save.
With this change the result for a path that changes or is deleted
between the drecurse traversal and the from_path() call should be the
same as the result for a path that changes or is deleted between index
and save.
Thanks to Yann Autissier <yann.autissier@gmail.com> for reporting the
problem and Gabriel Filion <gabster@lelutin.ca> for helping figure out
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Nov 2013 17:21:46 +0000 (11:21 -0600)]
_helpers.c: be smarter when converting unknown integer types to python.
For now assume that long long and unsigned long long are always the
the largest integer types, and convert unknown size integers to Python
via PyLong_FromUnsignedLongLong() or PyLong_FromLongLong(), depending
on the value's sign.
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Nov 2013 20:11:44 +0000 (14:11 -0600)]
Makefile: avoid using -Werror with clang for now.
Apparently the default compiler on OS X and FreeBSD is now clang, and
it doesn't like our CHECK_VALUE_FITS() definition. So for now, just
don't specify -Werror when we detect clang.
Thanks to Alexander Barton <alex@barton.de>, Thomas Klausner
<tk@giga.or.at>, Sebastian Schumb <sebastian@sebastians-site.de>, and
Zoran Zaric <zz@zoranzaric.de> for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Alexander Barton <alex@barton.de>
Ben Kelly [Sat, 9 Nov 2013 19:21:04 +0000 (14:21 -0500)]
Add 'tag -f' support.
Add a new flag, -f/--force, to bup tag. By analogy with 'git tag',
'bup tag -f foo commit' will create tag *foo* even if a tag with that
name already exists; the existing tag will be silently overwritten.
By analogy with 'rm -f', add 'bup tag -d -f'. All this does is
suppress the error when you attempt to 'bup tag -d' a tag that doesn't
exist; it is primarily for use in scripts that want to delete a tag
that may or may not exist in the first place without 'bup tag'
reporting a failure.
Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: add missing "tag" and wrap text in bup-tag.md;
put "-d" first in tag-cmd.py optspec line and bup-tag.md; adjust
commit message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 8 Nov 2013 00:33:43 +0000 (18:33 -0600)]
index.py: compute the index metadata offset more carefully.
Previously when reading bupindex.meta (or similar), bup computed the
metadata file offset for it's internal lookup table by just
subtracting the size of a re-encoding of the metadata from file.tell()
after the read. This works fine until we change the length of the
metadata representation (which we just did); then bup will get the
wrong length from the re-encoded data.
To fix this, do what we should have done originally and capture the
index metadata offset before reading the metadata via tell(), and use
that instead.
Thanks to Mark J Hewitt <mjh@idnet.com> for being the first (known)
victim and for helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 15 Oct 2013 19:20:10 +0000 (14:20 -0500)]
Add --map-user --map-group --map-uid and --map-gid options to restore.
Note that the usual metadata rules still appply, so a user or group
entry will normally take precedence over a uid/gid unless
--numeric-ids is specified. So if you want to map a uid, for example,
you'll either need --numeric-ids, or you'll need to clear the
user/group like this:
These options should also make it possible to recover from archives
that were broken as a result of our incorrect handling of
signed/unsigned stat values (recently fixed).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Gabriel Filion <gabster@lelutin.ca>
Rob Browning [Sat, 12 Oct 2013 18:59:07 +0000 (13:59 -0500)]
Treat stat fields correctly as signed, unsigned, or unspecified (cf. POSIX).
Patrick reported a problem with restore on Cygwin that was being
caused by bup's handling of unknown uid/gid values; it turns out that
Cygwin returns -1 in that situation:
http://www.cygwin.com/cygwin-ug-net/ntsec.html
"Special values of user and group ids"
However, bup didn't see -1; bup saw 4294967295, because it was
incorrectly converting some stat field values as unsigned when POSIX
says that they should be signed (or doesn't specify):
To fix the problem, convert the stat fields correctly -- using the
widest type available (PY_LONG_LONG), and add a new common metadata
record (_rec_tag_common_v2) to store them correctly. Test at runtime
for potential overflows that we can't rule out at compile time.
Store all of the fields (excepting the timestamp ns values, which we
already force to be non-negative) as signed vints because the overhead
should be negligible, and doing so means that we won't have to break
backward compatibility again if we ever decide we actually want to
store negative values.
Continue to read the legacy format as before, with the realization
that this may leave some existing archives broken. At the moment, the
only known case is the Cygwin "unknown uid/gid" problem cited above,
and to help with that and similar cases, we'll be adding
--map-{user,group,uid,gid} options to restore.
Thanks to Patrick Rouleau <prouleau72@gmail.com> for the initial
report leading to the deeper problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Gabriel Filion <gabster@lelutin.ca>
Thanks to Ben Kelly <btk@google.com> for tracking down the problem,
rather than Alexander Barton <alex@barton.de>.
Original message:
Author: Rob Browning <rlb@defaultvalue.org>
Date: Tue Nov 5 17:17:45 2013 -0600
Don't include atime when determining hardlink compatibility.
Don't include the atimes since we're comparing two paths stat()ed at
different times, and depending on the
platform/filesystem/mount-options, that may result in two different
atimes, even when the paths refer to the same file.
Thanks to Alexander Barton <alex@barton.de> for tracking down the
problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Gabriel Filion <gabster@lelutin.ca> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 5 Nov 2013 23:17:45 +0000 (17:17 -0600)]
Don't include atime when determining hardlink compatibility.
Don't include the atimes since we're comparing two paths stat()ed at
different times, and depending on the
platform/filesystem/mount-options, that may result in two different
atimes, even when the paths refer to the same file.
Thanks to Alexander Barton <alex@barton.de> for tracking down the
problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Gabriel Filion <gabster@lelutin.ca>
Rob Browning [Tue, 5 Nov 2013 22:29:53 +0000 (16:29 -0600)]
Add current-filesystem and path-filesystems to t/lib.sh; use to guard test.
Add current-filesystem which reports the current filesystem type, and
path-filesystems which reports all the filesystems back up to the
root, and use the latter to skip the "save --strip-path (no match)"
test whenever all the filesystems involved aren't the same type.
This issue was discovered while testing on
/some/ext4/tree/containing/a/btrfs/tree. When the test indexes/saves,
it picks up the metadata for the parents above the btrfs, which may
have (for example) linux +e attrs. Then when it tries to restore the
tree fully within the btrfs, the restore defers an error when it tries
to restore the +e, which isn't valid for btrfs.
Normally we can avoid this problem via --strip, but this particular
test is actually checking the behavior when --strip doesn't apply.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 5 Nov 2013 22:17:04 +0000 (16:17 -0600)]
_apply_linux_xattr_rec(): do nothing if no rec; fix restricted access test.
If the metadata object has no linux_xattr, don't do anything at all in
_apply_linux_xattr_rec(), and don't test the filesystem for xattr
support to determine whether or not to expect an xattr error, because
EACCES trumps the other errors.
This should be more efficient and fix a
test_apply_to_path_restricted_access() failure.
Thanks to Sebastian Schumb <sebastian@sebastians-site.de> for
reporting the problem and helping track down the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 4 Nov 2013 15:34:17 +0000 (09:34 -0600)]
test.sh: remove "wc -l" quoting until/unless we fix the broader issue.
On some platforms "wc -l" returns whitespace along with the count,
which breaks the one case where we quoted the result, i.e. "$(... | wc
-l"). For now, remove the quoting until/unless we decide to fix all
of the affected invocations.
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem on
NetBSD.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 27 Oct 2013 18:23:20 +0000 (13:23 -0500)]
Fix xattr tests and bugs revealed (fully remove set -e from test-meta.sh).
Fixing the xattr tests to run correctly pointed out that bup stops
trying to apply metadata after the first failure (i.e. after any
metadata record application fails). Change it so that bup always
tries every metadata record. Otherwise, bup may skip some forms of
metadata without any notification -- and it shouldn't. For example,
it shouldn't skip ACLs just because the current filesystem doesn't
support xattrs.
Fix two additional problems that were exposed by the changes described
above.
Check for EACCES when trying to apply an attr record, and properly
convert it to an ApplyError. This problem was revealed by
test_apply_to_path_restricted_access.
Check for EACCES when trying to read the current filesystem xattrs
during the process of applying the metadata xattrs, and when
encountered, convert it to an ApplyError.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 26 Oct 2013 20:03:12 +0000 (15:03 -0500)]
Fix the "set -e" problem in the test-meta.sh chattr tests.
Start fixing the "set -e" problem, beginning with the test-meta.sh
chattr tests, and add support to the test filesystem for options
needed by the chattr calls.
To allow us to produce a more digestable sequence of patches for the
broader "set -e" problem, temporarily establish BUP_SKIP_BROKEN_TESTS
which will do what it says when set, and until we finish repairing
everything, it must be set during any testing:
BUP_SKIP_BROKEN_TESTS=1 make check
The fundamental issue with "set -e" is that it has some unexpected
(but documented) semantics. For example:
set -e
foo() { false; rm -rf /; }
# Effectively harmless
foo
# Erase the filesystem
foo || true
If nothing else, this behavior appears to have been masking test
failures. Better to just avoid "set -e" altogether.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log